[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:09:13 dist-test-slave-4bwq krb5kdc[4781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:09:13 dist-test-slave-4bwq krb5kdc[4781](info): set up 2 sockets
Jun 20 20:09:13 dist-test-slave-4bwq krb5kdc[4781](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 20:09:14.060847 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:09:14 dist-test-slave-4bwq krb5kdc[4781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:09:14.238262 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42093
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:42093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:14.579963 4794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:14.580318 4794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:14.580477 4794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:14.588071 4794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:14.588243 4794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:14.588356 4794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:14.588461 4794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:14.588539 4794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:14.602531 4794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:42093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:14.603915 4794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:14.605197 4794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:14.617396 4799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:14.617489 4800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:14.617442 4802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:14.624203 4794 server_base.cc:1034] running on GCE node
I20260620 20:09:14.625012 4794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:14.626565 4794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:14.627828 4794 hybrid_clock.cc:648] HybridClock initialized: now 1781986154627789 us; error 50 us; skew 500 ppm
I20260620 20:09:14.628257 4794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:14 dist-test-slave-4bwq krb5kdc[4781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986154, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:14.631559 4794 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:09:14.633167 4794 webserver.cc:460] Webserver started at http://127.4.169.190:43041/ using document root <none> and password file <none>
I20260620 20:09:14.633912 4794 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:14.634059 4794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:14.634375 4794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:14.637392 4794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "a90b2eeb98594b3695493be149891fa1"
format_stamp: "Formatted at 2026-06-20 20:09:14 on dist-test-slave-4bwq"
server_key: "07c329f815d909647e3263575491d558"
server_key_iv: "062efb2fa1885cbe4dc01f6a9a17a155"
server_key_version: "encryptionkey@0"
I20260620 20:09:14.638160 4794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "a90b2eeb98594b3695493be149891fa1"
format_stamp: "Formatted at 2026-06-20 20:09:14 on dist-test-slave-4bwq"
server_key: "07c329f815d909647e3263575491d558"
server_key_iv: "062efb2fa1885cbe4dc01f6a9a17a155"
server_key_version: "encryptionkey@0"
I20260620 20:09:14.643540 4794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 20:09:14.647060 4809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:14.648355 4794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:14.648535 4794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "a90b2eeb98594b3695493be149891fa1"
format_stamp: "Formatted at 2026-06-20 20:09:14 on dist-test-slave-4bwq"
server_key: "07c329f815d909647e3263575491d558"
server_key_iv: "062efb2fa1885cbe4dc01f6a9a17a155"
server_key_version: "encryptionkey@0"
I20260620 20:09:14.648743 4794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:14.668509 4794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:14.672612 4794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:14.672901 4794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:14.692068 4794 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:42093
I20260620 20:09:14.692090 4860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:42093 every 8 connection(s)
I20260620 20:09:14.693567 4794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:14.696286 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 4794
I20260620 20:09:14.696637 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:14.698427 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:14.699193 4774 external_mini_cluster.cc:1428] Setting key 2de903d23ff3234e5418497d7ebbff72
I20260620 20:09:14.711124 4861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1: Bootstrap starting.
I20260620 20:09:14.714915 4861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:14.717425 4861 log.cc:824] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:14.720981 4861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1: No bootstrap required, opened a new log
I20260620 20:09:14.733440 4861 raft_consensus.cc:348] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90b2eeb98594b3695493be149891fa1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42093 } }
I20260620 20:09:14.733990 4861 raft_consensus.cc:374] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:14.734149 4861 raft_consensus.cc:729] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90b2eeb98594b3695493be149891fa1, State: Initialized, Role: FOLLOWER
Jun 20 20:09:14 dist-test-slave-4bwq krb5kdc[4781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:14.734898 4861 consensus_queue.cc:260] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90b2eeb98594b3695493be149891fa1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42093 } }
I20260620 20:09:14.735284 4861 raft_consensus.cc:388] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:14.735489 4861 raft_consensus.cc:482] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:14.735764 4861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:14.741262 4861 raft_consensus.cc:504] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90b2eeb98594b3695493be149891fa1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42093 } }
I20260620 20:09:14.742177 4861 leader_election.cc:302] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a90b2eeb98594b3695493be149891fa1; no voters:
I20260620 20:09:14.743501 4861 leader_election.cc:288] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:14.743695 4866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:14.746587 4861 sys_catalog.cc:564] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:14.747633 4866 raft_consensus.cc:686] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [term 1 LEADER]: Becoming Leader. State: Replica: a90b2eeb98594b3695493be149891fa1, State: Running, Role: LEADER
I20260620 20:09:14.748723 4866 consensus_queue.cc:237] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90b2eeb98594b3695493be149891fa1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42093 } }
I20260620 20:09:14.752938 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:14.703977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.4.169.190:42093)
0620 20:09:14.705047 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:14.705063 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:14.706466 (+ 1403us) server_negotiation.cc:407] Connection header received
0620 20:09:14.717511 (+ 11045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:14.717544 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:14.718203 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:14.718731 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:14.722033 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:14.724897 (+ 2864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:14.727175 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:14.728852 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:14.729099 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:14.736642 (+ 7543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:14.736667 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:14.736690 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:14.736782 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:14.743192 (+ 6410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:14.745076 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:14.745093 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:14.745108 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:14.745535 (+ 427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:14.746588 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:14.746593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:14.746597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:14.747653 (+ 1056us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:14.748129 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:14.749968 (+ 1839us) server_negotiation.cc:299] Negotiation successful
0620 20:09:14.750549 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":486,"threads_started":1}
I20260620 20:09:14.756220 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a90b2eeb98594b3695493be149891fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90b2eeb98594b3695493be149891fa1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42093 } } }
I20260620 20:09:14.757263 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:14.756852 4868 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a90b2eeb98594b3695493be149891fa1. Latest consensus state: current_term: 1 leader_uuid: "a90b2eeb98594b3695493be149891fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90b2eeb98594b3695493be149891fa1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42093 } } }
I20260620 20:09:14.759774 4868 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:14.761811 4874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:14.771390 4874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:14.784924 4874 catalog_manager.cc:1347] Generated new cluster ID: d79e2048463a4debaa28c85b45a314e9
I20260620 20:09:14.785126 4874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:14.823663 4874 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:14.825227 4874 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:14.843432 4874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a90b2eeb98594b3695493be149891fa1: Generated new TSK 0
I20260620 20:09:14.844755 4874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:09:14.892802 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42093
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:15.279323 4887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:15.279599 4887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:15.279771 4887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:15.287150 4887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:15.287314 4887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:15.287514 4887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:09:15.297459 4887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:15.298893 4887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:15.300268 4887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:15.314612 4892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:15.314652 4895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:15.314694 4893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:15.315492 4887 server_base.cc:1034] running on GCE node
I20260620 20:09:15.316062 4887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:15.316972 4887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:15.318161 4887 hybrid_clock.cc:648] HybridClock initialized: now 1781986155318123 us; error 36 us; skew 500 ppm
I20260620 20:09:15.318531 4887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:15 dist-test-slave-4bwq krb5kdc[4781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:15.321940 4887 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:09:15.323392 4887 webserver.cc:460] Webserver started at http://127.4.169.129:34747/ using document root <none> and password file <none>
I20260620 20:09:15.324018 4887 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:15.324111 4887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:15.324363 4887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:15.327152 4887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "7c0a63c3b1da4abab08552eb50a6f8eb"
format_stamp: "Formatted at 2026-06-20 20:09:15 on dist-test-slave-4bwq"
server_key: "fccc85c968c9a52078e9937911a3886a"
server_key_iv: "eda697ec6fbea74132ee9ee9c5465f3a"
server_key_version: "encryptionkey@0"
I20260620 20:09:15.327867 4887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "7c0a63c3b1da4abab08552eb50a6f8eb"
format_stamp: "Formatted at 2026-06-20 20:09:15 on dist-test-slave-4bwq"
server_key: "fccc85c968c9a52078e9937911a3886a"
server_key_iv: "eda697ec6fbea74132ee9ee9c5465f3a"
server_key_version: "encryptionkey@0"
I20260620 20:09:15.332852 4887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:09:15.336154 4902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:15.337464 4887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:09:15.337584 4887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "7c0a63c3b1da4abab08552eb50a6f8eb"
format_stamp: "Formatted at 2026-06-20 20:09:15 on dist-test-slave-4bwq"
server_key: "fccc85c968c9a52078e9937911a3886a"
server_key_iv: "eda697ec6fbea74132ee9ee9c5465f3a"
server_key_version: "encryptionkey@0"
I20260620 20:09:15.337733 4887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:15.361464 4887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:15.366194 4887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:15.366539 4887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:15.367535 4887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:15.368978 4887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:15.369048 4887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:15.369133 4887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:15.369174 4887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:15.404670 4887 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:41641
I20260620 20:09:15.404757 5014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:41641 every 8 connection(s)
I20260620 20:09:15.405979 4887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:09:15.413836 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 4887
I20260620 20:09:15.414077 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:09:15.414855 4774 external_mini_cluster.cc:1428] Setting key d6e6afe342e38f0a52c3b9533b89a240
Jun 20 20:09:15 dist-test-slave-4bwq krb5kdc[4781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:15.428073 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:15.408595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:49303 (local address 127.4.169.190:42093)
0620 20:09:15.409045 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:15.409050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:15.410034 (+ 984us) server_negotiation.cc:407] Connection header received
0620 20:09:15.412396 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:15.412402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:15.412501 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:15.412622 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:15.414510 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:15.415591 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:15.417590 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:15.418326 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:15.418548 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:15.422075 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:15.422080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:15.422083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:15.422114 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:15.424785 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:15.425562 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:15.425569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:15.425572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:15.425696 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:15.426125 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:15.426132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:15.426136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:15.426483 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:15.426625 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:15.427642 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0620 20:09:15.427809 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":234,"threads_started":1}
I20260620 20:09:15.429785 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:15.409170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42093 (local address 127.4.169.129:49303)
0620 20:09:15.409852 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:15.409898 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:15.412095 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:15.412865 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:15.412874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:15.413417 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:15.414301 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:15.414321 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:15.415748 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:15.415753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:15.417440 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:15.417452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:15.418612 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:15.418617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:15.418816 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:15.419526 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:15.419547 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:15.421850 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:15.424940 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:15.424950 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:15.424968 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:15.425436 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:15.425806 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:15.425811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:15.425815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:15.426015 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:15.426637 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:15.426646 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:15.427158 (+ 512us) client_negotiation.cc:769] Sending connection context
0620 20:09:15.427565 (+ 407us) client_negotiation.cc:240] Negotiation successful
0620 20:09:15.428219 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":270,"threads_started":1}
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:09:15.431490 5015 heartbeater.cc:347] Connected to a master server at 127.4.169.190:42093
I20260620 20:09:15.431840 5015 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:15.432593 5015 heartbeater.cc:511] Master 127.4.169.190:42093 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
I20260620 20:09:15.434983 4826 ts_manager.cc:194] Registered new tserver with Master: 7c0a63c3b1da4abab08552eb50a6f8eb (127.4.169.129:41641)
I20260620 20:09:15.436424 4826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:49303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:09:15.455667 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42093
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:15.790335 5021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:15.790660 5021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:15.790848 5021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:15.798255 5021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:15.798362 5021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:15.798588 5021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:09:15.809450 5021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:15.810837 5021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:15.812244 5021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:15.825433 5029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:15.825508 5027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:15.825627 5026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:15.826486 5021 server_base.cc:1034] running on GCE node
I20260620 20:09:15.827107 5021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:15.828063 5021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:15.829337 5021 hybrid_clock.cc:648] HybridClock initialized: now 1781986155829283 us; error 65 us; skew 500 ppm
I20260620 20:09:15.829696 5021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:15 dist-test-slave-4bwq krb5kdc[4781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:15.832978 5021 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:09:15.834604 5021 webserver.cc:460] Webserver started at http://127.4.169.130:42609/ using document root <none> and password file <none>
I20260620 20:09:15.835307 5021 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:15.835459 5021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:15.835773 5021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:15.838706 5021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "81a21fd71ba24d05a0a54e74e91807d4"
format_stamp: "Formatted at 2026-06-20 20:09:15 on dist-test-slave-4bwq"
server_key: "a24065a11a283d1151218b7e966edd22"
server_key_iv: "cce32bb8e1269f9de23f9aae4ae5a88d"
server_key_version: "encryptionkey@0"
I20260620 20:09:15.839474 5021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "81a21fd71ba24d05a0a54e74e91807d4"
format_stamp: "Formatted at 2026-06-20 20:09:15 on dist-test-slave-4bwq"
server_key: "a24065a11a283d1151218b7e966edd22"
server_key_iv: "cce32bb8e1269f9de23f9aae4ae5a88d"
server_key_version: "encryptionkey@0"
I20260620 20:09:15.844894 5021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:09:15.848830 5036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:15.850174 5021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:09:15.850364 5021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "81a21fd71ba24d05a0a54e74e91807d4"
format_stamp: "Formatted at 2026-06-20 20:09:15 on dist-test-slave-4bwq"
server_key: "a24065a11a283d1151218b7e966edd22"
server_key_iv: "cce32bb8e1269f9de23f9aae4ae5a88d"
server_key_version: "encryptionkey@0"
I20260620 20:09:15.850631 5021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:15.866318 5021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:15.870347 5021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:15.870661 5021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:15.871609 5021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:15.872967 5021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:15.873075 5021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:15.873216 5021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:15.873298 5021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:15.910060 5021 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:35061
I20260620 20:09:15.910101 5148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:35061 every 8 connection(s)
I20260620 20:09:15.911463 5021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:09:15.913614 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 5021
I20260620 20:09:15.913827 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:09:15.914320 4774 external_mini_cluster.cc:1428] Setting key 886a4f8b3002173b7b0ba154bc44f708
Jun 20 20:09:15 dist-test-slave-4bwq krb5kdc[4781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:09:15.933712 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:15.914335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:49243 (local address 127.4.169.190:42093)
0620 20:09:15.914577 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:15.914584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:15.915927 (+ 1343us) server_negotiation.cc:407] Connection header received
0620 20:09:15.917740 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:15.917749 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:15.917852 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:15.917966 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:15.919523 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:15.920540 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:15.922445 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:15.922889 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:15.923020 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:15.926509 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:15.926518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:15.926526 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:15.926574 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:15.930776 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:15.931544 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:15.931548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:15.931550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:15.931671 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:15.932037 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:15.932040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:15.932042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:15.932392 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:15.932501 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:15.933236 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 20:09:15.933436 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
I20260620 20:09:15.934850 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:15.914913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42093 (local address 127.4.169.130:49243)
0620 20:09:15.915735 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:15.915799 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:15.917464 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:15.918165 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:15.918173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:15.918764 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:15.919354 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:15.919371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:15.920699 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:15.920707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:15.922278 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:15.922290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:15.923119 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:15.923126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:15.923360 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:15.924055 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:15.924078 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:15.926225 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:15.930916 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:15.930923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:15.930938 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:15.931419 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:15.931766 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:15.931770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:15.931773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:15.931940 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:15.932502 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:15.932508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:15.932836 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 20:09:15.933158 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 20:09:15.933596 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":353,"threads_started":1}
I20260620 20:09:15.936388 5149 heartbeater.cc:347] Connected to a master server at 127.4.169.190:42093
I20260620 20:09:15.937135 5149 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:15.938000 5149 heartbeater.cc:511] Master 127.4.169.190:42093 requested a full tablet report, sending...
I20260620 20:09:15.939662 4826 ts_manager.cc:194] Registered new tserver with Master: 81a21fd71ba24d05a0a54e74e91807d4 (127.4.169.130:35061)
I20260620 20:09:15.940965 4826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:49243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:09:15.956730 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42093
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:16.287945 5154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:16.288275 5154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:16.288456 5154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:16.295928 5154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:16.296082 5154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:16.296279 5154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:09:16.305992 5154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:16.307493 5154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:16.308882 5154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:16.323675 5162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:16.323776 5160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:16.323705 5159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:16.327934 5154 server_base.cc:1034] running on GCE node
I20260620 20:09:16.328612 5154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:16.329691 5154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:16.330912 5154 hybrid_clock.cc:648] HybridClock initialized: now 1781986156330843 us; error 62 us; skew 500 ppm
I20260620 20:09:16.331251 5154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:16 dist-test-slave-4bwq krb5kdc[4781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986156, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:16.334600 5154 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:09:16.336129 5154 webserver.cc:460] Webserver started at http://127.4.169.131:40901/ using document root <none> and password file <none>
I20260620 20:09:16.336872 5154 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:16.337015 5154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:16.337328 5154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:16.340390 5154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "02178872542b411ba024c6b858353366"
format_stamp: "Formatted at 2026-06-20 20:09:16 on dist-test-slave-4bwq"
server_key: "b9ade3043130c11389d3c40600eaf557"
server_key_iv: "f9a450ee721d1d6ff3e54f485a57b170"
server_key_version: "encryptionkey@0"
I20260620 20:09:16.341157 5154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "02178872542b411ba024c6b858353366"
format_stamp: "Formatted at 2026-06-20 20:09:16 on dist-test-slave-4bwq"
server_key: "b9ade3043130c11389d3c40600eaf557"
server_key_iv: "f9a450ee721d1d6ff3e54f485a57b170"
server_key_version: "encryptionkey@0"
I20260620 20:09:16.346557 5154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:09:16.350360 5169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:16.351681 5154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:16.351864 5154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "02178872542b411ba024c6b858353366"
format_stamp: "Formatted at 2026-06-20 20:09:16 on dist-test-slave-4bwq"
server_key: "b9ade3043130c11389d3c40600eaf557"
server_key_iv: "f9a450ee721d1d6ff3e54f485a57b170"
server_key_version: "encryptionkey@0"
I20260620 20:09:16.352067 5154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:16.366052 5154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:16.370610 5154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:16.370911 5154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:16.371898 5154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:16.373346 5154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:16.373842 5154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:16.373987 5154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:16.374063 5154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:16.412614 5154 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:36795
I20260620 20:09:16.412703 5281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:36795 every 8 connection(s)
I20260620 20:09:16.413928 5154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:09:16.423697 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 5154
I20260620 20:09:16.423866 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:09:16.424382 4774 external_mini_cluster.cc:1428] Setting key 9387c92e1b1aeb39a3f9ee2c2ac0df7d
Jun 20 20:09:16 dist-test-slave-4bwq krb5kdc[4781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986156, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:16.435132 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:16.416496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:50595 (local address 127.4.169.190:42093)
0620 20:09:16.416691 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:16.416695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:16.417923 (+ 1228us) server_negotiation.cc:407] Connection header received
0620 20:09:16.419787 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:16.419792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:16.419894 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:16.420024 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:16.421486 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.422235 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.424301 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.424953 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.425101 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:16.429230 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:16.429235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:16.429238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:16.429272 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:16.432035 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.432796 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.432801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.432803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.432955 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.433407 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.433418 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.433421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.433675 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:16.433825 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:16.434561 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 20:09:16.434777 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:09:16.436120 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:16.417042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42093 (local address 127.4.169.131:50595)
0620 20:09:16.417757 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:16.417795 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:16.419521 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:16.420237 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:16.420246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:16.420751 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:16.421327 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:16.421345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.422448 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.422453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:16.424159 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:16.424172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.425164 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.425169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:16.425476 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:16.426289 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:16.426322 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:16.429019 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:16.432199 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:16.432208 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:16.432227 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:16.432659 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:16.433073 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:16.433078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:16.433082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:16.433284 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:16.433801 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:16.433809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:16.434164 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 20:09:16.434454 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 20:09:16.434994 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":295,"threads_started":1}
I20260620 20:09:16.437531 5282 heartbeater.cc:347] Connected to a master server at 127.4.169.190:42093
I20260620 20:09:16.437799 5282 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:16.438535 5282 heartbeater.cc:511] Master 127.4.169.190:42093 requested a full tablet report, sending...
I20260620 20:09:16.439383 5015 heartbeater.cc:503] Master 127.4.169.190:42093 was elected leader, sending a full tablet report...
I20260620 20:09:16.440145 4826 ts_manager.cc:194] Registered new tserver with Master: 02178872542b411ba024c6b858353366 (127.4.169.131:36795)
I20260620 20:09:16.441285 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:09:16.441305 4826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:50595
Jun 20 20:09:16 dist-test-slave-4bwq krb5kdc[4781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986156, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:09:16 dist-test-slave-4bwq krb5kdc[4781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986156, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:16.488665 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:16.472329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.4.169.129:41641)
0620 20:09:16.473404 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:16.473414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:16.473596 (+ 182us) server_negotiation.cc:407] Connection header received
0620 20:09:16.473827 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:16.473834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:16.474048 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:16.474242 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:16.475669 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.476700 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.477980 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.478665 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.478804 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:16.482787 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:16.482799 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:16.482814 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:16.482847 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:16.485458 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.486422 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.486434 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.486440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.486632 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.487020 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.487026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.487029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.487398 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:16.487704 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:16.488016 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 20:09:16.488291 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":289,"threads_started":1}
W20260620 20:09:16.489948 4930 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44530
Jun 20 20:09:16 dist-test-slave-4bwq krb5kdc[4781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:09:16 dist-test-slave-4bwq krb5kdc[4781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:16.534171 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:16.520694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.4.169.129:41641)
0620 20:09:16.520945 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:16.520952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:16.521227 (+ 275us) server_negotiation.cc:407] Connection header received
0620 20:09:16.521443 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:16.521447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:16.521538 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:16.521651 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:16.523055 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.523788 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.525043 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.525457 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.525587 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:16.529370 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:16.529375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:16.529378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:16.529410 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:16.531984 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.532556 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.532559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.532562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.532673 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.532982 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.532985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.532987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.533209 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:16.533380 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:16.533662 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 20:09:16.533822 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 20:09:16.535630 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 4887
I20260620 20:09:16.548791 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:16.536767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.4.169.129:41641)
0620 20:09:16.536979 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:16.536985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:16.537202 (+ 217us) server_negotiation.cc:407] Connection header received
0620 20:09:16.537400 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:16.537410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:16.537461 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:16.537559 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:16.538801 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.539492 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.540724 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.541152 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.541281 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:16.543505 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:16.543510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:16.543513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:16.543540 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:16.546214 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.546915 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.546922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.546925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.547052 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.547457 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.547461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.547464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.547706 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:16.547960 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:16.548249 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 20:09:16.548446 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:09:16.549484 5010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:16.631846 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 4887
I20260620 20:09:16.648077 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 5021
Jun 20 20:09:16 dist-test-slave-4bwq krb5kdc[4781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:09:16.663836 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:16.648935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.4.169.130:35061)
0620 20:09:16.649537 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:16.649544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:16.649569 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:09:16.649628 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:16.649635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:16.649848 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:16.649976 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:16.651228 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.652382 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.653596 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.654178 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.654273 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:16.658511 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:16.658520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:16.658533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:16.658570 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:16.661266 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.661875 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.661883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.661885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.662001 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.662436 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.662443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.662446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.662773 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:16.662985 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:16.663336 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 20:09:16.663545 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":332,"threads_started":1}
I20260620 20:09:16.665153 5144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:16.755326 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 5021
I20260620 20:09:16.777060 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 5154
Jun 20 20:09:16 dist-test-slave-4bwq krb5kdc[4781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:09:16.792973 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:16.778014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.4.169.131:36795)
0620 20:09:16.778685 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:16.778691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:16.778719 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:09:16.778778 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:16.778784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:16.778982 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:16.779145 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:16.780355 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.781302 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.782654 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:16.783193 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:16.783339 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:16.787416 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:16.787426 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:16.787439 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:16.787469 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:16.790039 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.790914 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.790923 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.790927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.791054 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:16.791415 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:16.791420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:16.791424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:16.791753 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:16.792081 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:16.792377 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 20:09:16.792633 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":377,"threads_started":1}
I20260620 20:09:16.794282 5277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:16.875782 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 5154
I20260620 20:09:16.891314 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 4794
I20260620 20:09:16.892493 4856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:16.994674 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 4794
2026-06-20T20:09:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4054 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:09:17 dist-test-slave-4bwq krb5kdc[5303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:09:17 dist-test-slave-4bwq krb5kdc[5303](info): set up 2 sockets
Jun 20 20:09:17 dist-test-slave-4bwq krb5kdc[5303](info): commencing operation
krb5kdc: starting...
W20260620 20:09:18.066951 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:09:18 dist-test-slave-4bwq krb5kdc[5303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:09:18.245276 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:32857
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:18.568490 5316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:18.568809 5316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:18.568993 5316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:18.576649 5316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:18.576761 5316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:18.576858 5316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:18.576959 5316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:18.577025 5316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:18.587019 5316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:18.588415 5316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:18.589745 5316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:18.601871 5324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:18.601912 5321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:18.601935 5322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:18.602708 5316 server_base.cc:1034] running on GCE node
I20260620 20:09:18.603477 5316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:18.604871 5316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:18.606105 5316 hybrid_clock.cc:648] HybridClock initialized: now 1781986158606061 us; error 58 us; skew 500 ppm
I20260620 20:09:18.606534 5316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:18 dist-test-slave-4bwq krb5kdc[5303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986158, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:18.612571 5316 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:09:18.614118 5316 webserver.cc:460] Webserver started at http://127.4.169.190:39185/ using document root <none> and password file <none>
I20260620 20:09:18.614866 5316 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:18.615008 5316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:18.615324 5316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:18.618211 5316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "ae23bcc62ee8489496f2d28d161f3a18"
format_stamp: "Formatted at 2026-06-20 20:09:18 on dist-test-slave-4bwq"
server_key: "3041a80338578cba7030d4b646f425c9"
server_key_iv: "2d7bc20bf641879a1e3eba3c134bbe1f"
server_key_version: "encryptionkey@0"
I20260620 20:09:18.619036 5316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "ae23bcc62ee8489496f2d28d161f3a18"
format_stamp: "Formatted at 2026-06-20 20:09:18 on dist-test-slave-4bwq"
server_key: "3041a80338578cba7030d4b646f425c9"
server_key_iv: "2d7bc20bf641879a1e3eba3c134bbe1f"
server_key_version: "encryptionkey@0"
I20260620 20:09:18.624382 5316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:09:18.628460 5331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:18.629837 5316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:09:18.630023 5316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "ae23bcc62ee8489496f2d28d161f3a18"
format_stamp: "Formatted at 2026-06-20 20:09:18 on dist-test-slave-4bwq"
server_key: "3041a80338578cba7030d4b646f425c9"
server_key_iv: "2d7bc20bf641879a1e3eba3c134bbe1f"
server_key_version: "encryptionkey@0"
I20260620 20:09:18.630220 5316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:18.661159 5316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:18.665879 5316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:18.666208 5316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:18.685387 5316 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:32857
I20260620 20:09:18.685406 5382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:32857 every 8 connection(s)
I20260620 20:09:18.686817 5316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:18.690630 5383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:18.693097 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 5316
I20260620 20:09:18.693329 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:18.694133 4774 external_mini_cluster.cc:1428] Setting key 1a6b8229127da6905a1afe9c6cde0fe3
I20260620 20:09:18.703488 5383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18: Bootstrap starting.
I20260620 20:09:18.707072 5383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:18.708137 5383 log.cc:824] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:09:18 dist-test-slave-4bwq krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:18.711123 5383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18: No bootstrap required, opened a new log
I20260620 20:09:18.718518 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:18.696564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.4.169.190:32857)
0620 20:09:18.697433 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:18.697446 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:18.697489 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:09:18.699346 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:18.699369 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:18.699809 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:18.700158 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:18.701998 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:18.703529 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:18.705477 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:18.706217 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:18.706505 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:18.710940 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:18.710948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:18.710964 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:18.711007 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:18.714276 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:18.715027 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:18.715039 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:18.715045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:18.715207 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:18.715608 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:18.715612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:18.715614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:18.716176 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:18.716511 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:18.716999 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 20:09:18.717430 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":429,"threads_started":1}
I20260620 20:09:18.719336 5383 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae23bcc62ee8489496f2d28d161f3a18" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32857 } }
I20260620 20:09:18.719726 5383 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:18.719866 5383 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae23bcc62ee8489496f2d28d161f3a18, State: Initialized, Role: FOLLOWER
I20260620 20:09:18.720409 5383 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae23bcc62ee8489496f2d28d161f3a18" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32857 } }
I20260620 20:09:18.720674 5383 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:18.720825 5383 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:18.720999 5383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:18.723385 5383 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae23bcc62ee8489496f2d28d161f3a18" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32857 } }
I20260620 20:09:18.723919 5383 leader_election.cc:302] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae23bcc62ee8489496f2d28d161f3a18; no voters:
I20260620 20:09:18.724540 5383 leader_election.cc:288] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:18.724720 5388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:18.726511 5388 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [term 1 LEADER]: Becoming Leader. State: Replica: ae23bcc62ee8489496f2d28d161f3a18, State: Running, Role: LEADER
I20260620 20:09:18.727049 5388 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae23bcc62ee8489496f2d28d161f3a18" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32857 } }
I20260620 20:09:18.727335 5383 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:18.730170 5389 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae23bcc62ee8489496f2d28d161f3a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae23bcc62ee8489496f2d28d161f3a18" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32857 } } }
I20260620 20:09:18.730574 5390 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae23bcc62ee8489496f2d28d161f3a18. Latest consensus state: current_term: 1 leader_uuid: "ae23bcc62ee8489496f2d28d161f3a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae23bcc62ee8489496f2d28d161f3a18" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32857 } } }
I20260620 20:09:18.731956 5389 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:18.732071 5390 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:18.739148 5396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:18.743649 5396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:18.752988 5396 catalog_manager.cc:1347] Generated new cluster ID: a2ab6fea19a7491d8b1b6b0b21a59b02
I20260620 20:09:18.753132 5396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:18.776813 5396 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:18.778949 5396 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:18.798107 5396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae23bcc62ee8489496f2d28d161f3a18: Generated new TSK 0
I20260620 20:09:18.799068 5396 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:09:18.850095 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:32857
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:19.181942 5409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:19.182228 5409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:19.182412 5409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:19.189625 5409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:19.189741 5409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:19.189980 5409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:09:19.200059 5409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:32857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:19.201385 5409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:19.202780 5409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:19.216305 5415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:19.216213 5414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:19.216262 5417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:19.217463 5409 server_base.cc:1034] running on GCE node
I20260620 20:09:19.218170 5409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:19.219233 5409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:19.220438 5409 hybrid_clock.cc:648] HybridClock initialized: now 1781986159220386 us; error 49 us; skew 500 ppm
I20260620 20:09:19.220773 5409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:19 dist-test-slave-4bwq krb5kdc[5303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:19.223935 5409 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:09:19.225369 5409 webserver.cc:460] Webserver started at http://127.4.169.129:38083/ using document root <none> and password file <none>
I20260620 20:09:19.226080 5409 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:19.226222 5409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:19.226586 5409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:19.229591 5409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "b6e38efb98934d3b8c734599405af42b"
format_stamp: "Formatted at 2026-06-20 20:09:19 on dist-test-slave-4bwq"
server_key: "eee0f61a4608ad4c6c03cdc96e722bf8"
server_key_iv: "d836dc8fd1aade44bbba78aece57aed6"
server_key_version: "encryptionkey@0"
I20260620 20:09:19.230379 5409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "b6e38efb98934d3b8c734599405af42b"
format_stamp: "Formatted at 2026-06-20 20:09:19 on dist-test-slave-4bwq"
server_key: "eee0f61a4608ad4c6c03cdc96e722bf8"
server_key_iv: "d836dc8fd1aade44bbba78aece57aed6"
server_key_version: "encryptionkey@0"
I20260620 20:09:19.236018 5409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:09:19.239569 5424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:19.240857 5409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:09:19.241031 5409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "b6e38efb98934d3b8c734599405af42b"
format_stamp: "Formatted at 2026-06-20 20:09:19 on dist-test-slave-4bwq"
server_key: "eee0f61a4608ad4c6c03cdc96e722bf8"
server_key_iv: "d836dc8fd1aade44bbba78aece57aed6"
server_key_version: "encryptionkey@0"
I20260620 20:09:19.241230 5409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:19.257716 5409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:19.261505 5409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:19.261796 5409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:19.262799 5409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:19.264091 5409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:19.264194 5409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:19.264340 5409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:19.264411 5409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:19.297566 5409 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:41697
I20260620 20:09:19.297662 5536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:41697 every 8 connection(s)
I20260620 20:09:19.299016 5409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:09:19.307004 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 5409
I20260620 20:09:19.307238 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:09:19.307976 4774 external_mini_cluster.cc:1428] Setting key c4cadc306c2287664629e7e3445801d2
Jun 20 20:09:19 dist-test-slave-4bwq krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:19.319726 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:19.301575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:46567 (local address 127.4.169.190:32857)
0620 20:09:19.302013 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:19.302017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:19.302929 (+ 912us) server_negotiation.cc:407] Connection header received
0620 20:09:19.304797 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:19.304802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:19.304891 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:19.305008 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:19.306524 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:19.307672 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:19.309673 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:19.310220 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:19.310352 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:19.313802 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:19.313808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:19.313819 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:19.313851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:19.316382 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:19.317349 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:19.317353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:19.317355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:19.317468 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:19.317964 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:19.317967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:19.317969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:19.318212 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:19.318317 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:19.319301 (+ 984us) server_negotiation.cc:299] Negotiation successful
0620 20:09:19.319475 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":240,"threads_started":1}
I20260620 20:09:19.321446 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:19.302003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:32857 (local address 127.4.169.129:46567)
0620 20:09:19.302770 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:19.302814 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:19.304519 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:19.305236 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:19.305244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:19.305737 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:19.306318 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:19.306336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:19.307840 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:19.307844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:19.309484 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:19.309499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:19.310478 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:19.310485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:19.310689 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:19.311368 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:19.311389 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:19.313555 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:19.316533 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:19.316545 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:19.316567 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:19.317219 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:19.317579 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:19.317587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:19.317594 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:19.317854 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:19.318339 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:19.318350 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:19.318885 (+ 535us) client_negotiation.cc:769] Sending connection context
0620 20:09:19.319199 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 20:09:19.319823 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":327,"threads_started":1}
I20260620 20:09:19.323315 5537 heartbeater.cc:347] Connected to a master server at 127.4.169.190:32857
I20260620 20:09:19.323647 5537 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:19.324404 5537 heartbeater.cc:511] Master 127.4.169.190:32857 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:09:19.326655 5348 ts_manager.cc:194] Registered new tserver with Master: b6e38efb98934d3b8c734599405af42b (127.4.169.129:41697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
I20260620 20:09:19.328742 5348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:46567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:09:19.349349 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:32857
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:19.672547 5543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:19.672837 5543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:19.672999 5543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:19.680325 5543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:19.680435 5543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:19.680601 5543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:09:19.690361 5543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:32857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:19.691672 5543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:19.693002 5543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:19.706311 5548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:19.706333 5551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:19.706252 5549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:19.707039 5543 server_base.cc:1034] running on GCE node
I20260620 20:09:19.707690 5543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:19.708668 5543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:19.709892 5543 hybrid_clock.cc:648] HybridClock initialized: now 1781986159709845 us; error 39 us; skew 500 ppm
I20260620 20:09:19.710203 5543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:19 dist-test-slave-4bwq krb5kdc[5303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:19.713346 5543 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:09:19.714967 5543 webserver.cc:460] Webserver started at http://127.4.169.130:40099/ using document root <none> and password file <none>
I20260620 20:09:19.715648 5543 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:19.715788 5543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:19.716203 5543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:19.719525 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "869b106dc7aa4348b41b894bd5ef1f85"
format_stamp: "Formatted at 2026-06-20 20:09:19 on dist-test-slave-4bwq"
server_key: "47fa600413742538b95435eb6f3262b2"
server_key_iv: "74245ec7288e4ac2d43bda293996c787"
server_key_version: "encryptionkey@0"
I20260620 20:09:19.720319 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "869b106dc7aa4348b41b894bd5ef1f85"
format_stamp: "Formatted at 2026-06-20 20:09:19 on dist-test-slave-4bwq"
server_key: "47fa600413742538b95435eb6f3262b2"
server_key_iv: "74245ec7288e4ac2d43bda293996c787"
server_key_version: "encryptionkey@0"
I20260620 20:09:19.725339 5543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:09:19.728969 5558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:19.730350 5543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:09:19.730585 5543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "869b106dc7aa4348b41b894bd5ef1f85"
format_stamp: "Formatted at 2026-06-20 20:09:19 on dist-test-slave-4bwq"
server_key: "47fa600413742538b95435eb6f3262b2"
server_key_iv: "74245ec7288e4ac2d43bda293996c787"
server_key_version: "encryptionkey@0"
I20260620 20:09:19.730801 5543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:19.754222 5543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:19.758976 5543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:19.759296 5543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:19.760341 5543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:19.761703 5543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:19.761826 5543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:19.761981 5543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:19.762054 5543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:19.795099 5543 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:39175
I20260620 20:09:19.795156 5670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:39175 every 8 connection(s)
I20260620 20:09:19.796494 5543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:09:19.805857 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 5543
I20260620 20:09:19.806058 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:09:19.806610 4774 external_mini_cluster.cc:1428] Setting key 6dd04a2e395e0f12937e1fc145184898
Jun 20 20:09:19 dist-test-slave-4bwq krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:19.817219 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:19.799287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:43255 (local address 127.4.169.190:32857)
0620 20:09:19.799463 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:19.799468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:19.800628 (+ 1160us) server_negotiation.cc:407] Connection header received
0620 20:09:19.802506 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:19.802511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:19.802579 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:19.802667 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:19.804096 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:19.804787 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:19.806412 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:19.806924 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:19.807064 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:19.811063 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:19.811070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:19.811074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:19.811119 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:19.813950 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:19.814719 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:19.814727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:19.814733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:19.814891 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:19.815279 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:19.815287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:19.815292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:19.815655 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:19.815829 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:19.816496 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 20:09:19.816753 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:09:19.817873 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:19.799756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:32857 (local address 127.4.169.130:43255)
0620 20:09:19.800459 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:19.800496 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:19.802197 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:19.802868 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:19.802876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:19.803389 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:19.803942 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:19.803959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:19.804907 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:19.804910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:19.806236 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:19.806248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:19.807148 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:19.807155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:19.807412 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:19.808069 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:19.808090 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:19.810813 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:19.814099 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:19.814106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:19.814119 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:19.814610 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:19.815005 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:19.815008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:19.815011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:19.815180 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:19.815778 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:19.815785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:19.816124 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 20:09:19.816363 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 20:09:19.816741 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":261,"threads_started":1}
I20260620 20:09:19.819342 5671 heartbeater.cc:347] Connected to a master server at 127.4.169.190:32857
I20260620 20:09:19.819686 5671 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:19.820580 5671 heartbeater.cc:511] Master 127.4.169.190:32857 requested a full tablet report, sending...
I20260620 20:09:19.822314 5348 ts_manager.cc:194] Registered new tserver with Master: 869b106dc7aa4348b41b894bd5ef1f85 (127.4.169.130:39175)
I20260620 20:09:19.823165 5348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:43255
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:09:19.847503 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:32857
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:20.169749 5676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:20.170014 5676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:20.170109 5676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:20.177263 5676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:20.177363 5676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:20.177474 5676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:09:20.187014 5676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:32857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:20.188197 5676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:20.189451 5676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:20.202457 5681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:20.202373 5684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:20.202430 5682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:20.203222 5676 server_base.cc:1034] running on GCE node
I20260620 20:09:20.203807 5676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:20.204893 5676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:20.206121 5676 hybrid_clock.cc:648] HybridClock initialized: now 1781986160206062 us; error 59 us; skew 500 ppm
I20260620 20:09:20.206524 5676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:20 dist-test-slave-4bwq krb5kdc[5303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986160, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:20.209619 5676 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:09:20.211215 5676 webserver.cc:460] Webserver started at http://127.4.169.131:43203/ using document root <none> and password file <none>
I20260620 20:09:20.211915 5676 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:20.212059 5676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:20.212394 5676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:20.215289 5676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "422dbffb6eeb41c2933eb2ba94d5f13d"
format_stamp: "Formatted at 2026-06-20 20:09:20 on dist-test-slave-4bwq"
server_key: "e9ddac5c2dae3db3446adc8c5bcf8bae"
server_key_iv: "8d872ef01d2b8b954f1e91951e214599"
server_key_version: "encryptionkey@0"
I20260620 20:09:20.215994 5676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "422dbffb6eeb41c2933eb2ba94d5f13d"
format_stamp: "Formatted at 2026-06-20 20:09:20 on dist-test-slave-4bwq"
server_key: "e9ddac5c2dae3db3446adc8c5bcf8bae"
server_key_iv: "8d872ef01d2b8b954f1e91951e214599"
server_key_version: "encryptionkey@0"
I20260620 20:09:20.220947 5676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:09:20.224579 5691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:20.225844 5676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:09:20.226015 5676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "422dbffb6eeb41c2933eb2ba94d5f13d"
format_stamp: "Formatted at 2026-06-20 20:09:20 on dist-test-slave-4bwq"
server_key: "e9ddac5c2dae3db3446adc8c5bcf8bae"
server_key_iv: "8d872ef01d2b8b954f1e91951e214599"
server_key_version: "encryptionkey@0"
I20260620 20:09:20.226220 5676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:20.249215 5676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:20.253437 5676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:20.253719 5676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:20.254799 5676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:20.256106 5676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:20.256214 5676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:20.256366 5676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:20.256439 5676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:20.289124 5676 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:40133
I20260620 20:09:20.289206 5803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:40133 every 8 connection(s)
I20260620 20:09:20.290441 5676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:09:20.294082 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 5676
I20260620 20:09:20.294234 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:09:20.294776 4774 external_mini_cluster.cc:1428] Setting key c3f78676078417996e40f6a671e5a184
Jun 20 20:09:20 dist-test-slave-4bwq krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986160, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:20.311535 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.292840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:44555 (local address 127.4.169.190:32857)
0620 20:09:20.293048 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.293054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.294737 (+ 1683us) server_negotiation.cc:407] Connection header received
0620 20:09:20.297010 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.297015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.297098 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.297190 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:20.299130 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.299871 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.302149 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.302600 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.302697 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.305908 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:20.305912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:20.305915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:20.305945 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:20.308555 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.309383 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.309388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.309391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.309547 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.309992 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.309996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.309999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.310316 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:20.310503 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.311131 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.311296 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:09:20.312479 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.293378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:32857 (local address 127.4.169.131:44555)
0620 20:09:20.294586 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:20.294629 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:20.296712 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:20.297412 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:20.297423 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:20.298154 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:20.298956 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.298977 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.300022 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.300029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.302013 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.302027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.302806 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.302816 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.303058 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.303659 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:20.303680 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:20.305737 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:20.308693 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:20.308702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:20.308717 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:20.309236 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:20.309654 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:20.309659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:20.309662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:20.309872 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:20.310496 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:20.310503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:20.310822 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 20:09:20.311044 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 20:09:20.311404 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":291,"threads_started":1}
I20260620 20:09:20.313787 5804 heartbeater.cc:347] Connected to a master server at 127.4.169.190:32857
I20260620 20:09:20.314043 5804 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:20.314749 5804 heartbeater.cc:511] Master 127.4.169.190:32857 requested a full tablet report, sending...
I20260620 20:09:20.316068 5348 ts_manager.cc:194] Registered new tserver with Master: 422dbffb6eeb41c2933eb2ba94d5f13d (127.4.169.131:40133)
I20260620 20:09:20.316826 5348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:44555
I20260620 20:09:20.322322 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:09:20.331811 5537 heartbeater.cc:503] Master 127.4.169.190:32857 was elected leader, sending a full tablet report...
I20260620 20:09:20.341502 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.328243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.4.169.190:32857)
0620 20:09:20.328422 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.328429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.328988 (+ 559us) server_negotiation.cc:407] Connection header received
0620 20:09:20.329119 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.329122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.329197 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.329306 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:20.330896 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.331573 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.333316 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.333836 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.333974 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.336465 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:20.336470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:20.336473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:20.336503 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:20.338913 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.339548 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.339555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.339559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.339698 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.340084 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.340089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.340093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.340387 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:20.340640 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.340951 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.341158 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:09:20.347909 5348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:09:20.354743 5348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:20.382674 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.367897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.4.169.131:40133)
0620 20:09:20.369777 (+ 1880us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.369787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.370319 (+ 532us) server_negotiation.cc:407] Connection header received
0620 20:09:20.370853 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.370862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.371176 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.371401 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:20.372538 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.374645 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.378180 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.380330 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.380468 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.380768 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.381692 (+ 924us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.381884 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1693,"thread_start_us":1539,"threads_started":1}
I20260620 20:09:20.382843 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.367591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:40133 (local address 127.0.0.1:58464)
0620 20:09:20.370165 (+ 2574us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:20.370215 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:20.370522 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:20.371784 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:20.371808 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:20.371858 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:20.372297 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.372313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.374870 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.374878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.377954 (+ 3076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.377974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.381003 (+ 3029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.381012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.381354 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.381425 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 20:09:20.381583 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 20:09:20.381783 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2320,"mutex_wait_us":1,"thread_start_us":2182,"threads_started":1}
I20260620 20:09:20.386857 5739 tablet_service.cc:1467] Processing CreateTablet for tablet 9d95916a76a24309b4ebe187a653f227 (DEFAULT_TABLE table=test-table [id=d636b42d50a04893a10381b3e750bd73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:20.388459 5739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d95916a76a24309b4ebe187a653f227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:20.388756 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.368876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.4.169.129:41697)
0620 20:09:20.370360 (+ 1484us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.370369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.372069 (+ 1700us) server_negotiation.cc:407] Connection header received
0620 20:09:20.372445 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.372456 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.372723 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.372943 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:20.374026 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.377334 (+ 3308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.383828 (+ 6494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.387654 (+ 3826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.387808 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.388048 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.388263 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.388435 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":1145,"threads_started":1}
I20260620 20:09:20.389356 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.368211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:41697 (local address 127.0.0.1:49648)
0620 20:09:20.370769 (+ 2558us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:20.370787 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:20.372209 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:20.373254 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:20.373262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:20.373294 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:20.373796 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.373809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.377540 (+ 3731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.377548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.383668 (+ 6120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.383681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.387882 (+ 4201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.387889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.388052 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.388088 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:09:20.388972 (+ 884us) client_negotiation.cc:240] Negotiation successful
0620 20:09:20.389098 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2427,"mutex_wait_us":1010,"thread_start_us":2288,"threads_started":1}
I20260620 20:09:20.389729 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.370902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:39175 (local address 127.0.0.1:36426)
0620 20:09:20.381951 (+ 11049us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:20.381969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:20.382097 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:20.383300 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:20.383304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:20.383329 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:20.383680 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.383689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.385481 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.385485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.386779 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.386789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.388256 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.388265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.388428 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.388465 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:09:20.388556 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 20:09:20.388675 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10914,"thread_start_us":10762,"threads_started":1}
I20260620 20:09:20.390051 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.370974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.4.169.130:39175)
0620 20:09:20.381562 (+ 10588us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.381572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.382089 (+ 517us) server_negotiation.cc:407] Connection header received
0620 20:09:20.382790 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.382799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.383098 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.383327 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:20.383895 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.385340 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.386931 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.388052 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.388205 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.388387 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.389042 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.389190 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10434,"thread_start_us":10345,"threads_started":1}
I20260620 20:09:20.392964 5472 tablet_service.cc:1467] Processing CreateTablet for tablet 9d95916a76a24309b4ebe187a653f227 (DEFAULT_TABLE table=test-table [id=d636b42d50a04893a10381b3e750bd73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:20.393533 5606 tablet_service.cc:1467] Processing CreateTablet for tablet 9d95916a76a24309b4ebe187a653f227 (DEFAULT_TABLE table=test-table [id=d636b42d50a04893a10381b3e750bd73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:20.394441 5472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d95916a76a24309b4ebe187a653f227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:20.395196 5606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d95916a76a24309b4ebe187a653f227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:20.403972 5822 tablet_bootstrap.cc:492] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d: Bootstrap starting.
I20260620 20:09:20.407287 5822 tablet_bootstrap.cc:654] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:20.408283 5822 log.cc:824] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:20.409521 5823 tablet_bootstrap.cc:492] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85: Bootstrap starting.
I20260620 20:09:20.411051 5825 tablet_bootstrap.cc:492] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b: Bootstrap starting.
I20260620 20:09:20.414120 5825 tablet_bootstrap.cc:654] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:20.414645 5823 tablet_bootstrap.cc:654] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:20.415429 5825 log.cc:824] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:20.416054 5823 log.cc:824] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:20.417459 5822 tablet_bootstrap.cc:492] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d: No bootstrap required, opened a new log
I20260620 20:09:20.417793 5822 ts_tablet_manager.cc:1397] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260620 20:09:20.420614 5825 tablet_bootstrap.cc:492] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b: No bootstrap required, opened a new log
I20260620 20:09:20.420871 5825 ts_tablet_manager.cc:1397] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 20:09:20.421928 5823 tablet_bootstrap.cc:492] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85: No bootstrap required, opened a new log
I20260620 20:09:20.422164 5823 ts_tablet_manager.cc:1397] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260620 20:09:20.428714 5825 raft_consensus.cc:348] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } }
I20260620 20:09:20.429163 5825 raft_consensus.cc:374] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:20.429250 5825 raft_consensus.cc:729] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e38efb98934d3b8c734599405af42b, State: Initialized, Role: FOLLOWER
I20260620 20:09:20.429219 5822 raft_consensus.cc:348] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } }
I20260620 20:09:20.429621 5822 raft_consensus.cc:374] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:20.429713 5822 raft_consensus.cc:729] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422dbffb6eeb41c2933eb2ba94d5f13d, State: Initialized, Role: FOLLOWER
I20260620 20:09:20.429955 5825 consensus_queue.cc:260] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } }
I20260620 20:09:20.430361 5822 consensus_queue.cc:260] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } }
I20260620 20:09:20.433853 5823 raft_consensus.cc:348] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } }
I20260620 20:09:20.434237 5804 heartbeater.cc:503] Master 127.4.169.190:32857 was elected leader, sending a full tablet report...
I20260620 20:09:20.434247 5823 raft_consensus.cc:374] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:20.434333 5823 raft_consensus.cc:729] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869b106dc7aa4348b41b894bd5ef1f85, State: Initialized, Role: FOLLOWER
I20260620 20:09:20.434926 5823 consensus_queue.cc:260] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } }
I20260620 20:09:20.435786 5825 ts_tablet_manager.cc:1428] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 20:09:20.436626 5822 ts_tablet_manager.cc:1428] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260620 20:09:20.437469 5671 heartbeater.cc:503] Master 127.4.169.190:32857 was elected leader, sending a full tablet report...
I20260620 20:09:20.438267 5823 ts_tablet_manager.cc:1428] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260620 20:09:20.532859 5828 raft_consensus.cc:482] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:20.533162 5828 raft_consensus.cc:504] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } }
I20260620 20:09:20.535584 5828 leader_election.cc:288] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 422dbffb6eeb41c2933eb2ba94d5f13d (127.4.169.131:40133), 869b106dc7aa4348b41b894bd5ef1f85 (127.4.169.130:39175)
W20260620 20:09:20.542022 5805 tablet.cc:2369] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:20.542668 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.535997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:40859 (local address 127.4.169.131:40133)
0620 20:09:20.536177 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.536181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.536899 (+ 718us) server_negotiation.cc:407] Connection header received
0620 20:09:20.536988 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.536993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.537068 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.537174 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:20.537903 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.538728 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.540776 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.541635 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.541789 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.541867 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.542248 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.542379 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:09:20.543408 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.535967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:40133 (local address 127.4.169.129:40859)
0620 20:09:20.536671 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:20.536693 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:20.536831 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:20.537340 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:20.537344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:20.537366 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:20.537717 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.537728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.538860 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.538866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.540604 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.540617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.541869 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.541875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.542042 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.542083 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:09:20.542905 (+ 822us) client_negotiation.cc:240] Negotiation successful
0620 20:09:20.543023 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":369,"threads_started":1}
I20260620 20:09:20.544003 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.536835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:39175 (local address 127.4.169.129:36801)
0620 20:09:20.537438 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:20.537455 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:20.537554 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:20.538463 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:20.538466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:20.538484 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:20.538747 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.538754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.539768 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.539773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.541401 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:20.541412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.543330 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.543336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:20.543479 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.543533 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 20:09:20.543623 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 20:09:20.543760 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":283,"threads_started":1}
I20260620 20:09:20.544060 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.537054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:36801 (local address 127.4.169.130:39175)
0620 20:09:20.537241 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.537246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.537544 (+ 298us) server_negotiation.cc:407] Connection header received
0620 20:09:20.537653 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.537658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.537732 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.537827 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:20.538960 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.539634 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.541553 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.543082 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.543245 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.543338 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.543670 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.543807 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:09:20.544560 5759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d95916a76a24309b4ebe187a653f227" candidate_uuid: "b6e38efb98934d3b8c734599405af42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" is_pre_election: true
I20260620 20:09:20.545099 5759 raft_consensus.cc:2454] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6e38efb98934d3b8c734599405af42b in term 0.
I20260620 20:09:20.545245 5626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d95916a76a24309b4ebe187a653f227" candidate_uuid: "b6e38efb98934d3b8c734599405af42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b106dc7aa4348b41b894bd5ef1f85" is_pre_election: true
I20260620 20:09:20.545732 5425 leader_election.cc:302] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422dbffb6eeb41c2933eb2ba94d5f13d, b6e38efb98934d3b8c734599405af42b; no voters:
I20260620 20:09:20.545773 5626 raft_consensus.cc:2454] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6e38efb98934d3b8c734599405af42b in term 0.
I20260620 20:09:20.546108 5828 raft_consensus.cc:2781] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:20.546262 5828 raft_consensus.cc:482] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:20.546409 5828 raft_consensus.cc:3037] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 0 FOLLOWER]: Advancing to term 1
W20260620 20:09:20.548445 5672 tablet.cc:2369] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:20.548679 5828 raft_consensus.cc:504] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } }
I20260620 20:09:20.549358 5828 leader_election.cc:288] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [CANDIDATE]: Term 1 election: Requested vote from peers 422dbffb6eeb41c2933eb2ba94d5f13d (127.4.169.131:40133), 869b106dc7aa4348b41b894bd5ef1f85 (127.4.169.130:39175)
I20260620 20:09:20.549733 5759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d95916a76a24309b4ebe187a653f227" candidate_uuid: "b6e38efb98934d3b8c734599405af42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d"
I20260620 20:09:20.549829 5626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d95916a76a24309b4ebe187a653f227" candidate_uuid: "b6e38efb98934d3b8c734599405af42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869b106dc7aa4348b41b894bd5ef1f85"
I20260620 20:09:20.549937 5759 raft_consensus.cc:3037] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:20.550046 5626 raft_consensus.cc:3037] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85 [term 0 FOLLOWER]: Advancing to term 1
W20260620 20:09:20.551321 5538 tablet.cc:2369] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:20.552343 5626 raft_consensus.cc:2454] T 9d95916a76a24309b4ebe187a653f227 P 869b106dc7aa4348b41b894bd5ef1f85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6e38efb98934d3b8c734599405af42b in term 1.
I20260620 20:09:20.552595 5759 raft_consensus.cc:2454] T 9d95916a76a24309b4ebe187a653f227 P 422dbffb6eeb41c2933eb2ba94d5f13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6e38efb98934d3b8c734599405af42b in term 1.
I20260620 20:09:20.552698 5425 leader_election.cc:302] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869b106dc7aa4348b41b894bd5ef1f85, b6e38efb98934d3b8c734599405af42b; no voters:
I20260620 20:09:20.553000 5828 raft_consensus.cc:2781] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:20.553431 5828 raft_consensus.cc:686] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [term 1 LEADER]: Becoming Leader. State: Replica: b6e38efb98934d3b8c734599405af42b, State: Running, Role: LEADER
I20260620 20:09:20.553866 5828 consensus_queue.cc:237] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } }
I20260620 20:09:20.558714 5347 catalog_manager.cc:5495] T 9d95916a76a24309b4ebe187a653f227 P b6e38efb98934d3b8c734599405af42b reported cstate change: term changed from 0 to 1, leader changed from <none> to b6e38efb98934d3b8c734599405af42b (127.4.169.129). New cstate: current_term: 1 leader_uuid: "b6e38efb98934d3b8c734599405af42b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422dbffb6eeb41c2933eb2ba94d5f13d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "869b106dc7aa4348b41b894bd5ef1f85" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6e38efb98934d3b8c734599405af42b" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41697 } health_report { overall_health: HEALTHY } } }
Jun 20 20:09:20 dist-test-slave-4bwq krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:20.621599 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.606447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.4.169.129:41697)
0620 20:09:20.606779 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.606787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.607089 (+ 302us) server_negotiation.cc:407] Connection header received
0620 20:09:20.607335 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.607340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.607401 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.607525 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:20.608969 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.609699 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.610996 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.611403 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.611532 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.615559 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:20.615574 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:20.615589 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:20.615620 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:20.618873 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.619575 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.619582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.619585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.619740 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.620119 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.620128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.620130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.620454 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:20.620764 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.621029 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.621262 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
Jun 20 20:09:20 dist-test-slave-4bwq krb5kdc[5303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:09:20 dist-test-slave-4bwq krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:20.667166 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.653366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.4.169.129:41697)
0620 20:09:20.653586 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.653593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.653955 (+ 362us) server_negotiation.cc:407] Connection header received
0620 20:09:20.654164 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.654168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.654230 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.654364 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:20.655655 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.656395 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.657629 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.658057 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.658191 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.662019 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:20.662024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:20.662027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:20.662059 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:20.664934 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.665524 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.665528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.665530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.665641 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.665971 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.665974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.665976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.666202 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:20.666480 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.666672 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.666831 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 20:09:20 dist-test-slave-4bwq krb5kdc[5303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:09:20 dist-test-slave-4bwq krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:20.714198 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.700493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44336 (local address 127.4.169.129:41697)
0620 20:09:20.700731 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.700737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.701033 (+ 296us) server_negotiation.cc:407] Connection header received
0620 20:09:20.701226 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.701230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.701310 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.701397 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:20.702740 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.703489 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.704645 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.705039 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.705130 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.709028 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:20.709033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:20.709036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:20.709068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:20.711919 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.712577 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.712580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.712583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.712695 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.713016 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.713019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.713021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.713268 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:20.713445 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.713677 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.713838 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 20:09:20.715924 5452 tablet_service.cc:2908] Rejecting scan request for tablet 9d95916a76a24309b4ebe187a653f227: Uninitialized: clean time has not yet been initialized
I20260620 20:09:20.719303 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 5409
I20260620 20:09:20.732543 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.720240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.4.169.129:41697)
0620 20:09:20.720447 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.720455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.720784 (+ 329us) server_negotiation.cc:407] Connection header received
0620 20:09:20.720964 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.720968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.721028 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.721116 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:20.722349 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.723315 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.724644 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.725041 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.725129 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.727539 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:20.727548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:20.727555 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:20.727596 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:20.730210 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.730896 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.730899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.730901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.731007 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.731307 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.731310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.731312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.731551 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:20.731715 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.731963 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.732145 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:09:20.733296 5532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:20.828997 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 5409
I20260620 20:09:20.846095 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 5543
Jun 20 20:09:20 dist-test-slave-4bwq krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:09:20.860458 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.847051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.4.169.130:39175)
0620 20:09:20.847219 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.847224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.847246 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:09:20.847322 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.847326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.847392 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.847473 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:20.848623 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.849393 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.850795 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.851186 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.851281 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.855296 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:20.855310 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:20.855326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:20.855359 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:20.858014 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.858760 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.858767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.858769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.858889 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.859198 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.859201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.859203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.859560 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:20.859786 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.860038 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.860196 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:09:20.861129 5666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:20.948815 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 5543
I20260620 20:09:20.965118 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 5676
Jun 20 20:09:20 dist-test-slave-4bwq krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:09:20.979070 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:20.965973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.4.169.131:40133)
0620 20:09:20.966149 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:20.966153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:20.966172 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:09:20.966344 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:20.966347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:20.966474 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:20.966562 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:20.967758 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.968455 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.969678 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:20.970040 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:20.970128 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:20.974109 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:20.974122 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:20.974138 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:20.974167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:20.976802 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.977379 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.977386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.977388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.977504 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:20.977793 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:20.977796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:20.977798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:20.978117 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:20.978321 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:20.978664 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 20:09:20.978809 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:09:20.979668 5799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:21.070376 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 5676
I20260620 20:09:21.087307 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 5316
I20260620 20:09:21.088538 5378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:21.194432 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 5316
2026-06-20T20:09:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4198 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:09:21 dist-test-slave-4bwq krb5kdc[5852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:09:21 dist-test-slave-4bwq krb5kdc[5852](info): set up 2 sockets
Jun 20 20:09:21 dist-test-slave-4bwq krb5kdc[5852](info): commencing operation
krb5kdc: starting...
W20260620 20:09:22.271015 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:09:22 dist-test-slave-4bwq krb5kdc[5852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
W20260620 20:09:23.686328 4774 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.169.190: real 1.267s user 0.000s sys 0.008s
I20260620 20:09:23.687743 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:36267
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:36267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:24.008955 5865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:24.009253 5865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:24.009395 5865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:24.016552 5865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:24.016654 5865 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:09:24.016731 5865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:24.016834 5865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:24.016918 5865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:24.017019 5865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:24.027002 5865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:36267
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:36267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:24.028294 5865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:24.029682 5865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:24.041381 5871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:24.041463 5870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:24.041432 5873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:24.042436 5865 server_base.cc:1034] running on GCE node
I20260620 20:09:24.043246 5865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:24.044569 5865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:24.045776 5865 hybrid_clock.cc:648] HybridClock initialized: now 1781986164045711 us; error 64 us; skew 500 ppm
I20260620 20:09:24.046100 5865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:24 dist-test-slave-4bwq krb5kdc[5852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986164, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:24.054266 5865 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:09:24.055816 5865 webserver.cc:460] Webserver started at http://127.4.169.190:37583/ using document root <none> and password file <none>
I20260620 20:09:24.056485 5865 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:24.056572 5865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:24.056849 5865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:24.059796 5865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "80054b47d6bc483c9a19beb573a7e8a2"
format_stamp: "Formatted at 2026-06-20 20:09:24 on dist-test-slave-4bwq"
server_key: "4b52fc3debf29ebed94cde5024635083"
server_key_iv: "537928ba542d01588e73016625d3cfa5"
server_key_version: "encryptionkey@0"
I20260620 20:09:24.060521 5865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "80054b47d6bc483c9a19beb573a7e8a2"
format_stamp: "Formatted at 2026-06-20 20:09:24 on dist-test-slave-4bwq"
server_key: "4b52fc3debf29ebed94cde5024635083"
server_key_iv: "537928ba542d01588e73016625d3cfa5"
server_key_version: "encryptionkey@0"
I20260620 20:09:24.065975 5865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:09:24.071378 5880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:24.073071 5865 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:09:24.073264 5865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "80054b47d6bc483c9a19beb573a7e8a2"
format_stamp: "Formatted at 2026-06-20 20:09:24 on dist-test-slave-4bwq"
server_key: "4b52fc3debf29ebed94cde5024635083"
server_key_iv: "537928ba542d01588e73016625d3cfa5"
server_key_version: "encryptionkey@0"
I20260620 20:09:24.073771 5865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:24.110822 5865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:24.116156 5865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:24.116385 5865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:24.139099 5865 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:36267
I20260620 20:09:24.139135 5941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:36267 every 8 connection(s)
I20260620 20:09:24.140519 5865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:24.144625 5942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:24.146770 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 5865
I20260620 20:09:24.146988 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:24.147938 4774 external_mini_cluster.cc:1428] Setting key 6178d617c1d8b494f366f47a0e497aa9
I20260620 20:09:24.158896 5942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2: Bootstrap starting.
Jun 20 20:09:24 dist-test-slave-4bwq krb5kdc[5852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:24.162199 5942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:24.163359 5942 log.cc:824] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:24.166338 5942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2: No bootstrap required, opened a new log
I20260620 20:09:24.169332 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:24.150190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.4.169.190:36267)
0620 20:09:24.151066 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:24.151078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:24.151124 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:09:24.152669 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:24.152689 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:24.153081 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:24.153453 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:24.154916 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.156234 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.157464 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.158143 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.158344 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:24.162212 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:24.162222 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:24.162236 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:24.162302 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:24.165473 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:24.166117 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:24.166130 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:24.166137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:24.166299 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:24.166704 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:24.166711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:24.166714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:24.167233 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:24.167548 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:24.167958 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 20:09:24.168366 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":412,"threads_started":1}
I20260620 20:09:24.175547 5942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80054b47d6bc483c9a19beb573a7e8a2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36267 } }
I20260620 20:09:24.175966 5942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:24.176077 5942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80054b47d6bc483c9a19beb573a7e8a2, State: Initialized, Role: FOLLOWER
I20260620 20:09:24.176657 5942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80054b47d6bc483c9a19beb573a7e8a2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36267 } }
I20260620 20:09:24.176934 5942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:24.177089 5942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:24.177273 5942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:24.179694 5942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80054b47d6bc483c9a19beb573a7e8a2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36267 } }
I20260620 20:09:24.180214 5942 leader_election.cc:302] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80054b47d6bc483c9a19beb573a7e8a2; no voters:
I20260620 20:09:24.180927 5942 leader_election.cc:288] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:24.181440 5947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:24.182346 5947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [term 1 LEADER]: Becoming Leader. State: Replica: 80054b47d6bc483c9a19beb573a7e8a2, State: Running, Role: LEADER
I20260620 20:09:24.182942 5947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80054b47d6bc483c9a19beb573a7e8a2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36267 } }
I20260620 20:09:24.183265 5942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:24.185722 5949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80054b47d6bc483c9a19beb573a7e8a2. Latest consensus state: current_term: 1 leader_uuid: "80054b47d6bc483c9a19beb573a7e8a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80054b47d6bc483c9a19beb573a7e8a2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36267 } } }
I20260620 20:09:24.186167 5949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:24.185781 5948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80054b47d6bc483c9a19beb573a7e8a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80054b47d6bc483c9a19beb573a7e8a2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36267 } } }
I20260620 20:09:24.186434 5948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:24.187765 5955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:24.192806 5955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:24.203526 5955 catalog_manager.cc:1347] Generated new cluster ID: caa94de3298d4ea9b737ac1285fc9150
I20260620 20:09:24.203680 5955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:24.212211 5955 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:24.213927 5955 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:24.224128 5955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2: Generated new TSK 0
I20260620 20:09:24.225113 5955 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:09:24.275652 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:36267
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:24.604895 5968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:24.605188 5968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:24.605284 5968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:24.613788 5968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:24.613974 5968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:24.614126 5968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:09:24.614297 5968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:09:24.625433 5968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.169.190:36267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:24.626783 5968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:24.628319 5968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:24.641604 5973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:24.641533 5974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:24.641752 5976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:24.642664 5968 server_base.cc:1034] running on GCE node
I20260620 20:09:24.643251 5968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:24.644160 5968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:24.645385 5968 hybrid_clock.cc:648] HybridClock initialized: now 1781986164645311 us; error 46 us; skew 500 ppm
I20260620 20:09:24.645677 5968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:24 dist-test-slave-4bwq krb5kdc[5852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986164, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:24.648898 5968 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:09:24.650449 5968 webserver.cc:460] Webserver started at http://127.4.169.129:34125/ using document root <none> and password file <none>
I20260620 20:09:24.651165 5968 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:24.651298 5968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:24.651631 5968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:24.654506 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "e0cabb282c3f4e16b086805651390732"
format_stamp: "Formatted at 2026-06-20 20:09:24 on dist-test-slave-4bwq"
server_key: "84073ecf516bc4b9e6d8993a41d44390"
server_key_iv: "12073d6db555941180eb155a085418fe"
server_key_version: "encryptionkey@0"
I20260620 20:09:24.655323 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "e0cabb282c3f4e16b086805651390732"
format_stamp: "Formatted at 2026-06-20 20:09:24 on dist-test-slave-4bwq"
server_key: "84073ecf516bc4b9e6d8993a41d44390"
server_key_iv: "12073d6db555941180eb155a085418fe"
server_key_version: "encryptionkey@0"
I20260620 20:09:24.660467 5968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 20:09:24.663949 5983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:24.665346 5968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:09:24.665518 5968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "e0cabb282c3f4e16b086805651390732"
format_stamp: "Formatted at 2026-06-20 20:09:24 on dist-test-slave-4bwq"
server_key: "84073ecf516bc4b9e6d8993a41d44390"
server_key_iv: "12073d6db555941180eb155a085418fe"
server_key_version: "encryptionkey@0"
I20260620 20:09:24.665731 5968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:24.682684 5968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:24.687057 5968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:24.687302 5968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:24.690024 5968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:24.690125 5968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:24.690272 5968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:24.690418 5968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:09:24 dist-test-slave-4bwq krb5kdc[5852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986164, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:24.713299 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:24.690512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:59491 (local address 127.4.169.190:36267)
0620 20:09:24.693121 (+ 2609us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:24.693130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:24.693160 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:09:24.694839 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:24.694848 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:24.694957 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:24.695090 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:24.697337 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.698628 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.701396 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.701838 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.701989 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:24.707173 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:24.707189 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:24.707192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:24.707223 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:24.709709 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:24.710485 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:24.710492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:24.710497 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:24.710626 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:24.711067 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:24.711072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:24.711075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:24.711345 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:24.711733 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:24.712679 (+ 946us) server_negotiation.cc:299] Negotiation successful
0620 20:09:24.712905 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2475,"thread_start_us":2384,"threads_started":1}
I20260620 20:09:24.714447 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:24.691290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36267 (local address 127.4.169.129:59491)
0620 20:09:24.692420 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:24.692484 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:24.694570 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:24.695367 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:24.695378 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:24.696174 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:24.697124 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:24.697159 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.698815 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.698823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:24.701089 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:24.701102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.702144 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.702155 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:24.702586 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:24.703422 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:24.703458 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:24.706952 (+ 3494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:24.709864 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:24.709873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:24.709889 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:24.710314 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:24.710745 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:24.710750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:24.710753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:24.710949 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:24.711690 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:24.711699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:24.712097 (+ 398us) client_negotiation.cc:769] Sending connection context
0620 20:09:24.712564 (+ 467us) client_negotiation.cc:240] Negotiation successful
0620 20:09:24.713135 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":469,"threads_started":1}
I20260620 20:09:24.738476 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:24.722105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:36575 (local address 127.4.169.190:36267)
0620 20:09:24.722320 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:24.722327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:24.723179 (+ 852us) server_negotiation.cc:407] Connection header received
0620 20:09:24.723511 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:24.723519 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:24.723618 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:24.723727 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:24.724981 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.726539 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.727854 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.728734 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.728899 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:24.731438 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:24.731448 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:24.731456 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:24.731503 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:24.735443 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:24.736260 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:24.736268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:24.736274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:24.736452 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:24.736935 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:24.736943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:24.736951 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:24.737286 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:24.737451 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:24.738011 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 20:09:24.738184 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:09:24.739210 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:24.722114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36267 (local address 127.4.169.129:36575)
0620 20:09:24.723043 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:24.723079 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:24.723302 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:24.723929 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:24.723940 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:24.724452 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:24.724821 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:24.724832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.726693 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.726697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:24.727715 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:24.727727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:24.728919 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:24.728923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:24.729052 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:24.729694 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:24.729710 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:24.731138 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:24.735651 (+ 4513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:24.735657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:24.735661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:24.736129 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:24.736588 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:24.736595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:24.736601 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:24.736788 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:24.737427 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:24.737435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:24.737721 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 20:09:24.737914 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 20:09:24.738413 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":1747,"threads_started":1}
I20260620 20:09:24.749300 5968 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:41759
I20260620 20:09:24.749320 6103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:41759 every 8 connection(s)
I20260620 20:09:24.750613 5968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:09:24.752549 6104 heartbeater.cc:347] Connected to a master server at 127.4.169.190:36267
I20260620 20:09:24.752799 6104 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:24.753445 6104 heartbeater.cc:511] Master 127.4.169.190:36267 requested a full tablet report, sending...
I20260620 20:09:24.753510 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 5968
I20260620 20:09:24.753720 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:09:24.754467 4774 external_mini_cluster.cc:1428] Setting key ae2d14e57b41ee93ccf2b3106bfe69ba
I20260620 20:09:24.755749 5897 ts_manager.cc:194] Registered new tserver with Master: e0cabb282c3f4e16b086805651390732 (127.4.169.129:41759)
I20260620 20:09:24.757364 5897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:59491
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:09:24.791829 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:36267
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:25.117017 6108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:25.117398 6108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:25.117601 6108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:25.124871 6108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:25.125030 6108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:25.125190 6108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:09:25.125344 6108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:09:25.135044 6108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.169.190:36267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:25.136485 6108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:25.137935 6108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:25.154045 6116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:25.154018 6114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:25.154778 6113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:25.155265 6108 server_base.cc:1034] running on GCE node
I20260620 20:09:25.155952 6108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:25.156963 6108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:25.158197 6108 hybrid_clock.cc:648] HybridClock initialized: now 1781986165158142 us; error 30 us; skew 500 ppm
I20260620 20:09:25.158519 6108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:25 dist-test-slave-4bwq krb5kdc[5852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:25.161644 6108 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:09:25.163033 6108 webserver.cc:460] Webserver started at http://127.4.169.130:41387/ using document root <none> and password file <none>
I20260620 20:09:25.163790 6108 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:25.163875 6108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:25.164126 6108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:25.167062 6108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "f39a81c01f1b439fa1ee8e08920d00fb"
format_stamp: "Formatted at 2026-06-20 20:09:25 on dist-test-slave-4bwq"
server_key: "a94d1d1e675728a471f38c5839822055"
server_key_iv: "ff273130cb6b5ef3d9f4def1ba333b51"
server_key_version: "encryptionkey@0"
I20260620 20:09:25.167809 6108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "f39a81c01f1b439fa1ee8e08920d00fb"
format_stamp: "Formatted at 2026-06-20 20:09:25 on dist-test-slave-4bwq"
server_key: "a94d1d1e675728a471f38c5839822055"
server_key_iv: "ff273130cb6b5ef3d9f4def1ba333b51"
server_key_version: "encryptionkey@0"
I20260620 20:09:25.173051 6108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:09:25.176352 6123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:25.177707 6108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 20:09:25.177832 6108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "f39a81c01f1b439fa1ee8e08920d00fb"
format_stamp: "Formatted at 2026-06-20 20:09:25 on dist-test-slave-4bwq"
server_key: "a94d1d1e675728a471f38c5839822055"
server_key_iv: "ff273130cb6b5ef3d9f4def1ba333b51"
server_key_version: "encryptionkey@0"
I20260620 20:09:25.178006 6108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:25.189847 6108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:25.194116 6108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:25.194332 6108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:25.196764 6108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:25.196839 6108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:25.196987 6108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:25.197041 6108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:09:25 dist-test-slave-4bwq krb5kdc[5852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:25.219324 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.197500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:37419 (local address 127.4.169.190:36267)
0620 20:09:25.197684 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.197690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.199148 (+ 1458us) server_negotiation.cc:407] Connection header received
0620 20:09:25.201077 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.201083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.201211 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.201322 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:25.203418 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.204445 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.207194 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.207721 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.207868 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.212440 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:25.212450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:25.212458 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:25.212510 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:25.215969 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.216809 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.216815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.216819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.216943 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.217426 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.217430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.217432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.217734 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:25.217907 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.218753 (+ 846us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.218972 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:09:25.221007 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.198107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36267 (local address 127.4.169.130:37419)
0620 20:09:25.198954 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:25.199002 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:25.200806 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:25.201584 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:25.201598 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:25.202348 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:25.203231 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.203262 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.204663 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.204667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.206972 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.206993 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.207984 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.207992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.208222 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.209077 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:25.209099 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:25.212143 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:25.216116 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:25.216126 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:25.216143 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:25.216671 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:25.217055 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:25.217060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:25.217064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:25.217265 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:25.217965 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:25.217971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:25.218324 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 20:09:25.218653 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 20:09:25.219249 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":397,"threads_started":1}
I20260620 20:09:25.250877 6108 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:40019
I20260620 20:09:25.250880 6242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:40019 every 8 connection(s)
I20260620 20:09:25.252650 6108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:09:25.254892 6243 heartbeater.cc:347] Connected to a master server at 127.4.169.190:36267
I20260620 20:09:25.255144 6243 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:25.256044 6243 heartbeater.cc:511] Master 127.4.169.190:36267 requested a full tablet report, sending...
I20260620 20:09:25.257586 5897 ts_manager.cc:194] Registered new tserver with Master: f39a81c01f1b439fa1ee8e08920d00fb (127.4.169.130:40019)
I20260620 20:09:25.257892 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.241076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36267 (local address 127.4.169.130:45491)
0620 20:09:25.241745 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:25.241772 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:25.242032 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:25.242696 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:25.242703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:25.243544 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:25.244115 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.244135 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.245736 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.245743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.247852 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.247875 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.248996 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.249001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.249107 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.249866 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:25.249908 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:25.251237 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:25.254763 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:25.254768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:25.254772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:25.255345 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:25.255818 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:25.255823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:25.255826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:25.255971 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:25.256763 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:25.256767 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:25.256961 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 20:09:25.257100 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:09:25.257434 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":378,"threads_started":1}
I20260620 20:09:25.258670 5897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:37419
I20260620 20:09:25.258744 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 6108
I20260620 20:09:25.258903 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:09:25.259526 4774 external_mini_cluster.cc:1428] Setting key 836737344d7d028e5bd9a67213a80a7f
I20260620 20:09:25.260751 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.241015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:45491 (local address 127.4.169.190:36267)
0620 20:09:25.241217 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.241225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.241881 (+ 656us) server_negotiation.cc:407] Connection header received
0620 20:09:25.242225 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.242234 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.242326 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.242443 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:25.244333 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.245515 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.248069 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.248672 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.248749 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.251428 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:25.251440 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:25.251444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:25.251476 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:25.254615 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.255509 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.255514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.255517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.255678 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.256117 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.256122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.256125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.256561 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:25.256703 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.258268 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.260444 (+ 2176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:09:25.308234 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:36267
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:25.639042 6247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:25.639304 6247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:25.639487 6247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:25.646726 6247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:25.646829 6247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:25.646971 6247 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:09:25.647092 6247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:09:25.656752 6247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.169.190:36267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:25.658033 6247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:25.659416 6247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:25.672168 6252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:25.672303 6253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:25.672556 6255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:25.673521 6247 server_base.cc:1034] running on GCE node
I20260620 20:09:25.674124 6247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:25.675099 6247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:25.676333 6247 hybrid_clock.cc:648] HybridClock initialized: now 1781986165676267 us; error 51 us; skew 500 ppm
I20260620 20:09:25.676653 6247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:25 dist-test-slave-4bwq krb5kdc[5852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:25.679741 6247 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:09:25.681209 6247 webserver.cc:460] Webserver started at http://127.4.169.131:36277/ using document root <none> and password file <none>
I20260620 20:09:25.681957 6247 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:25.682044 6247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:25.682344 6247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:25.685252 6247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "e0836be128054b62a424eedbe620f17c"
format_stamp: "Formatted at 2026-06-20 20:09:25 on dist-test-slave-4bwq"
server_key: "30c74fbbf1be4d651caaa06724de7d75"
server_key_iv: "314d4a993453007a287387c7c8a91e11"
server_key_version: "encryptionkey@0"
I20260620 20:09:25.686077 6247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "e0836be128054b62a424eedbe620f17c"
format_stamp: "Formatted at 2026-06-20 20:09:25 on dist-test-slave-4bwq"
server_key: "30c74fbbf1be4d651caaa06724de7d75"
server_key_iv: "314d4a993453007a287387c7c8a91e11"
server_key_version: "encryptionkey@0"
I20260620 20:09:25.691509 6247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:09:25.694999 6262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:25.696313 6247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:25.696491 6247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "e0836be128054b62a424eedbe620f17c"
format_stamp: "Formatted at 2026-06-20 20:09:25 on dist-test-slave-4bwq"
server_key: "30c74fbbf1be4d651caaa06724de7d75"
server_key_iv: "314d4a993453007a287387c7c8a91e11"
server_key_version: "encryptionkey@0"
I20260620 20:09:25.696682 6247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:25.722802 6247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:25.727208 6247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:25.727504 6247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:25.730157 6247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:25.730237 6247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:25.730413 6247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:25.730486 6247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 20:09:25 dist-test-slave-4bwq krb5kdc[5852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:25.756089 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.731691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36267 (local address 127.4.169.131:49341)
0620 20:09:25.732608 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:25.732658 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:25.734711 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:25.735732 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:25.735741 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:25.736747 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:25.737858 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.737894 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.739443 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.739451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.741518 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.741538 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.742904 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.742909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.743222 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.744092 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:25.744139 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:25.747051 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:25.751672 (+ 4621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:25.751681 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:25.751702 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:25.752349 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:25.752877 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:25.752882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:25.752886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:25.753094 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:25.753838 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:25.753846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:25.754285 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 20:09:25.754634 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 20:09:25.755088 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":391,"threads_started":1}
I20260620 20:09:25.757118 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.731098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:49341 (local address 127.4.169.190:36267)
0620 20:09:25.731301 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.731308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.732810 (+ 1502us) server_negotiation.cc:407] Connection header received
0620 20:09:25.735139 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.735148 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.735299 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.735435 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:25.738128 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.739261 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.741755 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.742434 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.742582 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.747309 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:25.747318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:25.747324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:25.747387 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:25.751491 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.752490 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.752499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.752506 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.752730 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.753208 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.753216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.753222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.753675 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:25.753847 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.756627 (+ 2780us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.756887 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:09:25.760176 6104 heartbeater.cc:503] Master 127.4.169.190:36267 was elected leader, sending a full tablet report...
I20260620 20:09:25.781965 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.765882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36267 (local address 127.4.169.131:47853)
0620 20:09:25.766738 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:25.766764 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:25.766949 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:25.767422 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:25.767430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:25.768265 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:25.768727 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.768742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.769833 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.769841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.772124 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.772143 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.773072 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.773082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.773323 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.774120 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:25.774150 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:25.775535 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:25.779053 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:25.779065 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:25.779068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:25.779665 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:25.780103 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:25.780111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:25.780116 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:25.780313 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:25.780916 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:25.780924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:25.781211 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 20:09:25.781373 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 20:09:25.781652 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":432,"threads_started":1}
I20260620 20:09:25.781987 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.765669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:47853 (local address 127.4.169.190:36267)
0620 20:09:25.765819 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.765823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.766872 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 20:09:25.767094 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.767111 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.767253 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.767377 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:25.768891 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.769698 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.772298 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.772845 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.773002 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.775705 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:25.775718 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:25.775721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:25.775750 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:25.778885 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.779796 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.779804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.779809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.779972 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.780460 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.780465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.780468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.780760 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:25.780974 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.781479 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.781683 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:09:25.785667 6247 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:45065
I20260620 20:09:25.785668 6381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:45065 every 8 connection(s)
I20260620 20:09:25.787036 6247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:09:25.788698 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 6247
I20260620 20:09:25.788868 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:09:25.789292 4774 external_mini_cluster.cc:1428] Setting key 1aed6591db94674f36808a4d0ef4575f
I20260620 20:09:25.789095 6382 heartbeater.cc:347] Connected to a master server at 127.4.169.190:36267
I20260620 20:09:25.790067 6382 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:25.790848 6382 heartbeater.cc:511] Master 127.4.169.190:36267 requested a full tablet report, sending...
I20260620 20:09:25.792428 5897 ts_manager.cc:194] Registered new tserver with Master: e0836be128054b62a424eedbe620f17c (127.4.169.131:45065)
I20260620 20:09:25.793170 5897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:49341
I20260620 20:09:25.804334 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:09:25 dist-test-slave-4bwq krb5kdc[5852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:09:25 dist-test-slave-4bwq krb5kdc[5852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:25.848552 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.835126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.4.169.190:36267)
0620 20:09:25.835337 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.835347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.835552 (+ 205us) server_negotiation.cc:407] Connection header received
0620 20:09:25.835737 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.835741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.835805 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.835890 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:25.837238 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.837986 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.839297 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.839704 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.839796 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.843707 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:25.843712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:25.843715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:25.843746 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:25.846125 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.846770 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.846774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.846777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.846885 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:25.847268 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:25.847271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:25.847273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:25.847536 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:25.847700 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.848010 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.848166 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:09:25.852630 5897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:09:25.859408 5897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:25.887698 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.872139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.4.169.131:45065)
0620 20:09:25.875074 (+ 2935us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.875087 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.875814 (+ 727us) server_negotiation.cc:407] Connection header received
0620 20:09:25.875942 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.875953 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.876313 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.876540 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:25.877816 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.880052 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.884474 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.886193 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.886330 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.886656 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.887142 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.887330 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2709,"thread_start_us":2502,"threads_started":1}
I20260620 20:09:25.888075 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.871705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:45065 (local address 127.0.0.1:56296)
0620 20:09:25.875557 (+ 3852us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:25.875573 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:25.875680 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:25.876927 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:25.876939 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:25.876979 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:25.877522 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.877538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.880278 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.880288 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.884325 (+ 4037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.884335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.886581 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.886587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.886868 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.886939 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 20:09:25.887103 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 20:09:25.887376 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3713,"mutex_wait_us":27,"thread_start_us":3591,"threads_started":1}
I20260620 20:09:25.891116 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.871594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:40019 (local address 127.0.0.1:46998)
0620 20:09:25.872809 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:25.872857 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:25.873063 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:25.883657 (+ 10594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:25.883662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:25.883687 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:25.884092 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.884104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.886140 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.886145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.888606 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.888620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.890284 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.890290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.890490 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.890522 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:09:25.890610 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 20:09:25.890740 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":105,"thread_start_us":933,"threads_started":1}
I20260620 20:09:25.891943 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.871744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.4.169.130:40019)
0620 20:09:25.882980 (+ 11236us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.882993 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.883033 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:09:25.883147 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.883155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.883413 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.884326 (+ 913us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:25.884519 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.885990 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.888759 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.890071 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.890233 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.890468 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.891484 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.891656 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11040,"thread_start_us":10907,"threads_started":1}
I20260620 20:09:25.892045 6312 tablet_service.cc:1467] Processing CreateTablet for tablet 9a840ca62dd847debb26ce1dedd86b59 (DEFAULT_TABLE table=test-table [id=f84d1f6aa4614815ac04e72cd5c0ca8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:25.892988 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.875635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:41759 (local address 127.0.0.1:58018)
0620 20:09:25.883400 (+ 7765us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:25.883425 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:25.883568 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:25.884813 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:25.884817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:25.884841 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:25.885227 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.885237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.887009 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.887015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.889120 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.889130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.891499 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.891507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.891687 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.891716 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:09:25.891783 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:09:25.891909 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7603,"mutex_wait_us":20,"thread_start_us":7437,"threads_started":1}
I20260620 20:09:25.893496 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a840ca62dd847debb26ce1dedd86b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:25.893937 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.875968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.4.169.129:41759)
0620 20:09:25.884184 (+ 8216us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.884190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.884213 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:09:25.884340 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.884347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.884616 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.884815 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:25.885441 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.886854 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.889265 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.891175 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.893209 (+ 2034us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.893407 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.893480 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.893640 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8085,"thread_start_us":7867,"threads_started":1}
I20260620 20:09:25.895211 6173 tablet_service.cc:1467] Processing CreateTablet for tablet 9a840ca62dd847debb26ce1dedd86b59 (DEFAULT_TABLE table=test-table [id=f84d1f6aa4614815ac04e72cd5c0ca8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:25.896224 6173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a840ca62dd847debb26ce1dedd86b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:25.897572 6034 tablet_service.cc:1467] Processing CreateTablet for tablet 9a840ca62dd847debb26ce1dedd86b59 (DEFAULT_TABLE table=test-table [id=f84d1f6aa4614815ac04e72cd5c0ca8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:25.898905 6034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a840ca62dd847debb26ce1dedd86b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:25.907685 6401 tablet_bootstrap.cc:492] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: Bootstrap starting.
I20260620 20:09:25.910970 6401 tablet_bootstrap.cc:654] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:25.911136 6402 tablet_bootstrap.cc:492] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: Bootstrap starting.
I20260620 20:09:25.912408 6401 log.cc:824] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:25.913700 6404 tablet_bootstrap.cc:492] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: Bootstrap starting.
I20260620 20:09:25.914757 6402 tablet_bootstrap.cc:654] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:25.915576 6401 tablet_bootstrap.cc:492] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: No bootstrap required, opened a new log
I20260620 20:09:25.915860 6401 ts_tablet_manager.cc:1397] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 20:09:25.916371 6402 log.cc:824] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:25.917604 6404 tablet_bootstrap.cc:654] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:25.918794 6404 log.cc:824] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:25.919507 6402 tablet_bootstrap.cc:492] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: No bootstrap required, opened a new log
I20260620 20:09:25.919762 6402 ts_tablet_manager.cc:1397] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:09:25.921630 6404 tablet_bootstrap.cc:492] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: No bootstrap required, opened a new log
I20260620 20:09:25.922294 6404 ts_tablet_manager.cc:1397] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 20:09:25.923709 6401 raft_consensus.cc:348] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } }
I20260620 20:09:25.924045 6401 raft_consensus.cc:374] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:25.924144 6401 raft_consensus.cc:729] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39a81c01f1b439fa1ee8e08920d00fb, State: Initialized, Role: FOLLOWER
I20260620 20:09:25.924731 6401 consensus_queue.cc:260] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } }
I20260620 20:09:25.927469 6243 heartbeater.cc:503] Master 127.4.169.190:36267 was elected leader, sending a full tablet report...
I20260620 20:09:25.927991 6401 ts_tablet_manager.cc:1428] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260620 20:09:25.930426 6402 raft_consensus.cc:348] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } }
I20260620 20:09:25.931242 6402 raft_consensus.cc:374] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:25.931341 6402 raft_consensus.cc:729] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0836be128054b62a424eedbe620f17c, State: Initialized, Role: FOLLOWER
I20260620 20:09:25.931931 6402 consensus_queue.cc:260] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } }
I20260620 20:09:25.933141 6382 heartbeater.cc:503] Master 127.4.169.190:36267 was elected leader, sending a full tablet report...
I20260620 20:09:25.933660 6402 ts_tablet_manager.cc:1428] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
I20260620 20:09:25.934540 6404 raft_consensus.cc:348] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } }
I20260620 20:09:25.934935 6404 raft_consensus.cc:374] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:25.935029 6404 raft_consensus.cc:729] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cabb282c3f4e16b086805651390732, State: Initialized, Role: FOLLOWER
I20260620 20:09:25.935669 6404 consensus_queue.cc:260] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } }
I20260620 20:09:25.937206 6404 ts_tablet_manager.cc:1428] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260620 20:09:25.948939 6408 raft_consensus.cc:482] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:25.949261 6408 raft_consensus.cc:504] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } }
I20260620 20:09:25.951041 6408 leader_election.cc:288] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f39a81c01f1b439fa1ee8e08920d00fb (127.4.169.130:40019), e0cabb282c3f4e16b086805651390732 (127.4.169.129:41759)
I20260620 20:09:25.957489 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.951302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:41759 (local address 127.4.169.131:58863)
0620 20:09:25.951590 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:25.951611 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:25.951745 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:25.952173 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:25.952178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:25.952210 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:25.952429 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.952439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.953735 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.953739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.955774 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.955792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.956939 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.956943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.957090 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.957114 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:09:25.957194 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:09:25.957290 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260620 20:09:25.957629 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.951573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:58863 (local address 127.4.169.129:41759)
0620 20:09:25.951835 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.951844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.951866 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:09:25.951942 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.951947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.952029 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.952124 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:25.952564 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.953601 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.955932 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.956787 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.956941 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.957017 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.957234 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.957365 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:09:25.958690 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.951381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:40019 (local address 127.4.169.131:51309)
0620 20:09:25.952116 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:25.952141 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:25.952305 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:25.952622 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:25.952625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:25.952644 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:25.952918 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.952924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.954096 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.954100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.956485 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:25.956496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.958027 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.958032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:25.958143 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.958164 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:09:25.958246 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:09:25.958327 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"spinlock_wait_cycles":31488,"thread_start_us":440,"threads_started":1}
I20260620 20:09:25.958732 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:25.951373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:51309 (local address 127.4.169.130:40019)
0620 20:09:25.951594 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:25.951603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:25.952251 (+ 648us) server_negotiation.cc:407] Connection header received
0620 20:09:25.952430 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:25.952434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:25.952511 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:25.952605 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:25.953060 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.953975 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.956637 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:25.957742 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:25.957926 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:25.958006 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:25.958296 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 20:09:25.958462 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 20:09:25.958675 6059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a840ca62dd847debb26ce1dedd86b59" candidate_uuid: "e0836be128054b62a424eedbe620f17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cabb282c3f4e16b086805651390732" is_pre_election: true
I20260620 20:09:25.959303 6059 raft_consensus.cc:2454] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0836be128054b62a424eedbe620f17c in term 0.
I20260620 20:09:25.959735 6193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a840ca62dd847debb26ce1dedd86b59" candidate_uuid: "e0836be128054b62a424eedbe620f17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" is_pre_election: true
I20260620 20:09:25.960170 6193 raft_consensus.cc:2454] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0836be128054b62a424eedbe620f17c in term 0.
I20260620 20:09:25.960171 6266 leader_election.cc:302] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0836be128054b62a424eedbe620f17c, e0cabb282c3f4e16b086805651390732; no voters:
I20260620 20:09:25.960561 6408 raft_consensus.cc:2781] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:25.960702 6408 raft_consensus.cc:482] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:25.960806 6408 raft_consensus.cc:3037] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:25.963038 6408 raft_consensus.cc:504] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } }
I20260620 20:09:25.963647 6408 leader_election.cc:288] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [CANDIDATE]: Term 1 election: Requested vote from peers f39a81c01f1b439fa1ee8e08920d00fb (127.4.169.130:40019), e0cabb282c3f4e16b086805651390732 (127.4.169.129:41759)
I20260620 20:09:25.963956 6193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a840ca62dd847debb26ce1dedd86b59" candidate_uuid: "e0836be128054b62a424eedbe620f17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39a81c01f1b439fa1ee8e08920d00fb"
I20260620 20:09:25.964067 6059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a840ca62dd847debb26ce1dedd86b59" candidate_uuid: "e0836be128054b62a424eedbe620f17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cabb282c3f4e16b086805651390732"
I20260620 20:09:25.964143 6193 raft_consensus.cc:3037] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:25.964291 6059 raft_consensus.cc:3037] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:25.966312 6193 raft_consensus.cc:2454] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0836be128054b62a424eedbe620f17c in term 1.
I20260620 20:09:25.966475 6059 raft_consensus.cc:2454] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0836be128054b62a424eedbe620f17c in term 1.
I20260620 20:09:25.966727 6263 leader_election.cc:302] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0836be128054b62a424eedbe620f17c, f39a81c01f1b439fa1ee8e08920d00fb; no voters:
I20260620 20:09:25.967047 6408 raft_consensus.cc:2781] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:25.967839 6408 raft_consensus.cc:686] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 1 LEADER]: Becoming Leader. State: Replica: e0836be128054b62a424eedbe620f17c, State: Running, Role: LEADER
I20260620 20:09:25.968397 6408 consensus_queue.cc:237] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } }
I20260620 20:09:25.972752 5897 catalog_manager.cc:5495] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c reported cstate change: term changed from 0 to 1, leader changed from <none> to e0836be128054b62a424eedbe620f17c (127.4.169.131). New cstate: current_term: 1 leader_uuid: "e0836be128054b62a424eedbe620f17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } health_report { overall_health: UNKNOWN } } }
W20260620 20:09:26.002957 6105 tablet.cc:2369] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:26.017160 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.011785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.4.169.131:45065)
0620 20:09:26.011953 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.011958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.012052 (+ 94us) server_negotiation.cc:407] Connection header received
0620 20:09:26.012389 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.012394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.012490 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.012616 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:26.012967 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.013662 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.015018 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.015428 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.015522 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.015999 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:26.016507 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:26.016664 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.016805 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.016866 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:09:26.017910 6244 tablet.cc:2369] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:26.024914 6193 raft_consensus.cc:1261] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [term 1 FOLLOWER]: Refusing update from remote peer e0836be128054b62a424eedbe620f17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:26.024914 6059 raft_consensus.cc:1261] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Refusing update from remote peer e0836be128054b62a424eedbe620f17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:26.025789 6411 consensus_queue.cc:1035] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:26.026110 6408 consensus_queue.cc:1035] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:26.038589 6416 mvcc.cc:205] Tried to move back new op lower bound from 7299015336028897280 to 7299015335810502656. Current Snapshot: MvccSnapshot[applied={T|T < 7299015336028897280}]
I20260620 20:09:26.038751 6415 mvcc.cc:205] Tried to move back new op lower bound from 7299015336028897280 to 7299015335810502656. Current Snapshot: MvccSnapshot[applied={T|T < 7299015336028897280}]
I20260620 20:09:26.064062 5896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:26.064378 5896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:26.068138 5896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2: Sending DeleteTablet for 3 replicas of tablet 9a840ca62dd847debb26ce1dedd86b59
I20260620 20:09:26.069330 6173 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a840ca62dd847debb26ce1dedd86b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:26 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:46998
I20260620 20:09:26.069425 6312 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a840ca62dd847debb26ce1dedd86b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:26 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:56296
I20260620 20:09:26.069952 6034 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a840ca62dd847debb26ce1dedd86b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:26 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:58018
I20260620 20:09:26.073319 5896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:09:26.074294 5896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:26.084251 6033 tablet_service.cc:1467] Processing CreateTablet for tablet d2e21b09a24c4626ac3dc46de6c872b9 (DEFAULT_TABLE table=test-table [id=45b590aa2eb144f18e5ee2dfb5d68b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:26.084370 6311 tablet_service.cc:1467] Processing CreateTablet for tablet d2e21b09a24c4626ac3dc46de6c872b9 (DEFAULT_TABLE table=test-table [id=45b590aa2eb144f18e5ee2dfb5d68b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:26.084892 6033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e21b09a24c4626ac3dc46de6c872b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:26.085139 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e21b09a24c4626ac3dc46de6c872b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:26.087345 6172 tablet_service.cc:1467] Processing CreateTablet for tablet d2e21b09a24c4626ac3dc46de6c872b9 (DEFAULT_TABLE table=test-table [id=45b590aa2eb144f18e5ee2dfb5d68b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:26.087900 6172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e21b09a24c4626ac3dc46de6c872b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:26.092968 6404 tablet_bootstrap.cc:492] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: Bootstrap starting.
I20260620 20:09:26.094483 6402 tablet_bootstrap.cc:492] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: Bootstrap starting.
I20260620 20:09:26.094664 6404 tablet_bootstrap.cc:654] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:26.096167 6402 tablet_bootstrap.cc:654] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:26.097431 6401 tablet_bootstrap.cc:492] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: Bootstrap starting.
I20260620 20:09:26.099784 6401 tablet_bootstrap.cc:654] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:26.111130 6424 tablet_replica.cc:330] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: stopping tablet replica
I20260620 20:09:26.111189 6402 tablet_bootstrap.cc:492] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: No bootstrap required, opened a new log
I20260620 20:09:26.111332 6402 ts_tablet_manager.cc:1397] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.001s
I20260620 20:09:26.112252 6424 raft_consensus.cc:2229] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:26.112654 6402 raft_consensus.cc:348] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.112892 6402 raft_consensus.cc:374] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:26.112978 6402 raft_consensus.cc:729] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0836be128054b62a424eedbe620f17c, State: Initialized, Role: FOLLOWER
I20260620 20:09:26.113237 6402 consensus_queue.cc:260] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.113859 6402 ts_tablet_manager.cc:1428] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 20:09:26.112982 6424 raft_consensus.cc:2258] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:26.115127 6401 tablet_bootstrap.cc:492] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: No bootstrap required, opened a new log
I20260620 20:09:26.115311 6401 ts_tablet_manager.cc:1397] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20260620 20:09:26.116304 6401 raft_consensus.cc:348] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.116593 6401 raft_consensus.cc:374] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:26.116724 6401 raft_consensus.cc:729] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39a81c01f1b439fa1ee8e08920d00fb, State: Initialized, Role: FOLLOWER
I20260620 20:09:26.116986 6404 tablet_bootstrap.cc:492] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: No bootstrap required, opened a new log
I20260620 20:09:26.117149 6404 ts_tablet_manager.cc:1397] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20260620 20:09:26.117065 6401 consensus_queue.cc:260] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.117275 6424 ts_tablet_manager.cc:1905] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:26.117648 6401 ts_tablet_manager.cc:1428] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:09:26.118227 6426 tablet_replica.cc:330] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: stopping tablet replica
I20260620 20:09:26.119293 6426 raft_consensus.cc:2229] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:26.119771 6426 raft_consensus.cc:2258] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:26.118462 6404 raft_consensus.cc:348] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.120746 6404 raft_consensus.cc:374] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:26.120846 6404 raft_consensus.cc:729] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cabb282c3f4e16b086805651390732, State: Initialized, Role: FOLLOWER
I20260620 20:09:26.121193 6404 consensus_queue.cc:260] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.121747 6404 ts_tablet_manager.cc:1428] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 20:09:26.121920 6426 ts_tablet_manager.cc:1905] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:26.123550 6425 tablet_replica.cc:330] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: stopping tablet replica
I20260620 20:09:26.124643 6425 raft_consensus.cc:2229] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:09:26.125396 6425 raft_consensus.cc:2258] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:26.128028 6425 ts_tablet_manager.cc:1905] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:26.129580 6426 ts_tablet_manager.cc:1918] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:09:26.129774 6426 log.cc:1196] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/9a840ca62dd847debb26ce1dedd86b59
I20260620 20:09:26.130321 6426 ts_tablet_manager.cc:1939] T 9a840ca62dd847debb26ce1dedd86b59 P e0cabb282c3f4e16b086805651390732: Deleting consensus metadata
I20260620 20:09:26.132284 5884 catalog_manager.cc:4841] TS e0cabb282c3f4e16b086805651390732 (127.4.169.129:41759): tablet 9a840ca62dd847debb26ce1dedd86b59 (table test-table [id=f84d1f6aa4614815ac04e72cd5c0ca8d]) successfully deleted
I20260620 20:09:26.133287 6424 ts_tablet_manager.cc:1918] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:09:26.133476 6424 log.cc:1196] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/9a840ca62dd847debb26ce1dedd86b59
I20260620 20:09:26.134003 6424 ts_tablet_manager.cc:1939] T 9a840ca62dd847debb26ce1dedd86b59 P f39a81c01f1b439fa1ee8e08920d00fb: Deleting consensus metadata
I20260620 20:09:26.136214 5881 catalog_manager.cc:4841] TS f39a81c01f1b439fa1ee8e08920d00fb (127.4.169.130:40019): tablet 9a840ca62dd847debb26ce1dedd86b59 (table test-table [id=f84d1f6aa4614815ac04e72cd5c0ca8d]) successfully deleted
I20260620 20:09:26.137048 6425 ts_tablet_manager.cc:1918] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:09:26.137238 6425 log.cc:1196] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/9a840ca62dd847debb26ce1dedd86b59
I20260620 20:09:26.137705 6425 ts_tablet_manager.cc:1939] T 9a840ca62dd847debb26ce1dedd86b59 P e0836be128054b62a424eedbe620f17c: Deleting consensus metadata
I20260620 20:09:26.139236 5884 catalog_manager.cc:4841] TS e0836be128054b62a424eedbe620f17c (127.4.169.131:45065): tablet 9a840ca62dd847debb26ce1dedd86b59 (table test-table [id=f84d1f6aa4614815ac04e72cd5c0ca8d]) successfully deleted
I20260620 20:09:26.240954 6407 raft_consensus.cc:482] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:26.241183 6407 raft_consensus.cc:504] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.242823 6407 leader_election.cc:288] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cabb282c3f4e16b086805651390732 (127.4.169.129:41759), e0836be128054b62a424eedbe620f17c (127.4.169.131:45065)
I20260620 20:09:26.251133 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.243065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:41759 (local address 127.4.169.130:60977)
0620 20:09:26.243689 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:26.243711 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:26.243901 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:26.244562 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:26.244569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:26.244603 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:26.245064 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.245080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.246567 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.246572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.248032 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.248046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.250250 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.250257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.250485 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.250520 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:09:26.250614 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 20:09:26.250771 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":358,"threads_started":1}
I20260620 20:09:26.251839 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.243180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:60977 (local address 127.4.169.129:41759)
0620 20:09:26.243386 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.243394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.243789 (+ 395us) server_negotiation.cc:407] Connection header received
0620 20:09:26.244100 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.244109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.244213 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.244294 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:26.245262 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.246422 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.248183 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.249847 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.250000 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.250078 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.251405 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.251547 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:09:26.252358 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.243967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:47541 (local address 127.4.169.131:45065)
0620 20:09:26.244136 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.244142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.245759 (+ 1617us) server_negotiation.cc:407] Connection header received
0620 20:09:26.245940 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.245944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.246003 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.246110 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:26.246758 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.247914 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.249677 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.250998 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.251139 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.251208 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.251911 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.252058 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:09:26.252985 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.243840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:45065 (local address 127.4.169.130:47541)
0620 20:09:26.245643 (+ 1803us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:26.245658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:26.245787 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:26.246122 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:26.246127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:26.246148 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:26.246552 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.246564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.248051 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.248056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.249349 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.249361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.251502 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.251508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.251692 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.251729 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:09:26.252532 (+ 803us) client_negotiation.cc:240] Negotiation successful
0620 20:09:26.252661 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1707,"thread_start_us":1531,"threads_started":1}
I20260620 20:09:26.253412 6059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e21b09a24c4626ac3dc46de6c872b9" candidate_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cabb282c3f4e16b086805651390732" is_pre_election: true
I20260620 20:09:26.253674 6059 raft_consensus.cc:2454] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f39a81c01f1b439fa1ee8e08920d00fb in term 0.
I20260620 20:09:26.253717 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e21b09a24c4626ac3dc46de6c872b9" candidate_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0836be128054b62a424eedbe620f17c" is_pre_election: true
I20260620 20:09:26.254123 6333 raft_consensus.cc:2454] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f39a81c01f1b439fa1ee8e08920d00fb in term 0.
I20260620 20:09:26.254376 6127 leader_election.cc:302] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0cabb282c3f4e16b086805651390732, f39a81c01f1b439fa1ee8e08920d00fb; no voters:
I20260620 20:09:26.254848 6407 raft_consensus.cc:2781] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:26.254966 6407 raft_consensus.cc:482] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:26.255060 6407 raft_consensus.cc:3037] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:26.258271 6407 raft_consensus.cc:504] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.259056 6407 leader_election.cc:288] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [CANDIDATE]: Term 1 election: Requested vote from peers e0cabb282c3f4e16b086805651390732 (127.4.169.129:41759), e0836be128054b62a424eedbe620f17c (127.4.169.131:45065)
I20260620 20:09:26.259763 6059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e21b09a24c4626ac3dc46de6c872b9" candidate_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cabb282c3f4e16b086805651390732"
I20260620 20:09:26.259848 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e21b09a24c4626ac3dc46de6c872b9" candidate_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0836be128054b62a424eedbe620f17c"
I20260620 20:09:26.259984 6059 raft_consensus.cc:3037] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:26.260056 6333 raft_consensus.cc:3037] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:26.263262 6059 raft_consensus.cc:2454] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f39a81c01f1b439fa1ee8e08920d00fb in term 1.
I20260620 20:09:26.263321 6333 raft_consensus.cc:2454] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f39a81c01f1b439fa1ee8e08920d00fb in term 1.
I20260620 20:09:26.263883 6127 leader_election.cc:302] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0cabb282c3f4e16b086805651390732, f39a81c01f1b439fa1ee8e08920d00fb; no voters:
I20260620 20:09:26.264247 6407 raft_consensus.cc:2781] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:26.264810 6407 raft_consensus.cc:686] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 1 LEADER]: Becoming Leader. State: Replica: f39a81c01f1b439fa1ee8e08920d00fb, State: Running, Role: LEADER
I20260620 20:09:26.265316 6407 consensus_queue.cc:237] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.269533 5896 catalog_manager.cc:5495] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb reported cstate change: term changed from 0 to 1, leader changed from <none> to f39a81c01f1b439fa1ee8e08920d00fb (127.4.169.130). New cstate: current_term: 1 leader_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } health_report { overall_health: UNKNOWN } } }
W20260620 20:09:26.290103 6383 tablet.cc:2369] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:09:26 dist-test-slave-4bwq krb5kdc[5852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986164, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:26.333586 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.316972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.4.169.190:36267)
0620 20:09:26.317189 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.317198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.317828 (+ 630us) server_negotiation.cc:407] Connection header received
0620 20:09:26.318028 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.318033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.318135 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.318259 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:26.319428 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.320552 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.322245 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.322920 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.323061 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.326656 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:26.326663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:26.326668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:26.326715 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:26.330505 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.331268 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.331274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.331277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.331467 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.331837 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.331843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.331847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.332232 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:26.332387 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.332940 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.333242 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":72}
I20260620 20:09:26.333619 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.317002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36267 (local address 127.0.0.1:40422)
0620 20:09:26.317695 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:26.317713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:26.317880 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:26.318292 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:26.318295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:26.318992 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:26.319255 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.319264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.321097 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.321101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.322084 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.322096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.323101 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.323106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.323225 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.323986 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:26.324003 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:26.326453 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:26.330691 (+ 4238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:26.330707 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:26.330714 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:26.331129 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:26.331573 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:26.331577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:26.331580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:26.331724 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:26.332442 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:26.332454 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:26.332664 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 20:09:26.332831 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 20:09:26.333093 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":354,"threads_started":1}
I20260620 20:09:26.339267 5896 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:40422:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:09:26.340323 5896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:26.347442 6034 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5098b9d9b94a3a8f6c27e9e10db17f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4181d15ae9a4728af30afa5dbf451c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:09:26.348017 6034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5098b9d9b94a3a8f6c27e9e10db17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:26.347966 6173 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5098b9d9b94a3a8f6c27e9e10db17f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4181d15ae9a4728af30afa5dbf451c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:09:26.348448 6173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5098b9d9b94a3a8f6c27e9e10db17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:26.350961 6312 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5098b9d9b94a3a8f6c27e9e10db17f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4181d15ae9a4728af30afa5dbf451c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:09:26.351503 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5098b9d9b94a3a8f6c27e9e10db17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:26.354223 6404 tablet_bootstrap.cc:492] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: Bootstrap starting.
I20260620 20:09:26.355917 6404 tablet_bootstrap.cc:654] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:26.356976 6401 tablet_bootstrap.cc:492] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: Bootstrap starting.
I20260620 20:09:26.359062 6401 tablet_bootstrap.cc:654] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:26.360257 6402 tablet_bootstrap.cc:492] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: Bootstrap starting.
I20260620 20:09:26.361905 6402 tablet_bootstrap.cc:654] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:26.365912 6402 tablet_bootstrap.cc:492] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: No bootstrap required, opened a new log
I20260620 20:09:26.366219 6402 ts_tablet_manager.cc:1397] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:09:26.367434 6402 raft_consensus.cc:348] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.367973 6402 raft_consensus.cc:374] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:26.368009 6404 tablet_bootstrap.cc:492] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: No bootstrap required, opened a new log
I20260620 20:09:26.368139 6402 raft_consensus.cc:729] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0836be128054b62a424eedbe620f17c, State: Initialized, Role: FOLLOWER
I20260620 20:09:26.368175 6404 ts_tablet_manager.cc:1397] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260620 20:09:26.368602 6402 consensus_queue.cc:260] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.368909 6404 raft_consensus.cc:348] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.369164 6404 raft_consensus.cc:374] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:26.369277 6404 raft_consensus.cc:729] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cabb282c3f4e16b086805651390732, State: Initialized, Role: FOLLOWER
I20260620 20:09:26.369525 6404 consensus_queue.cc:260] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.369585 6408 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.369944 6408 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:26.369954 6409 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.370290 6409 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:26.371225 6404 ts_tablet_manager.cc:1428] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 20:09:26.373211 6402 ts_tablet_manager.cc:1428] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260620 20:09:26.373549 6401 tablet_bootstrap.cc:492] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: No bootstrap required, opened a new log
I20260620 20:09:26.373742 6401 ts_tablet_manager.cc:1397] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20260620 20:09:26.374433 6401 raft_consensus.cc:348] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.374688 6401 raft_consensus.cc:374] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:26.374775 6401 raft_consensus.cc:729] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39a81c01f1b439fa1ee8e08920d00fb, State: Initialized, Role: FOLLOWER
I20260620 20:09:26.375046 6401 consensus_queue.cc:260] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.375484 6407 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.375759 6407 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:26.377075 6401 ts_tablet_manager.cc:1428] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 20:09:26.462229 6408 raft_consensus.cc:482] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:26.462495 6408 raft_consensus.cc:504] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.463294 6408 leader_election.cc:288] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cabb282c3f4e16b086805651390732 (127.4.169.129:41759), f39a81c01f1b439fa1ee8e08920d00fb (127.4.169.130:40019)
I20260620 20:09:26.463872 6059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5098b9d9b94a3a8f6c27e9e10db17f" candidate_uuid: "e0836be128054b62a424eedbe620f17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cabb282c3f4e16b086805651390732" is_pre_election: true
I20260620 20:09:26.463946 6193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5098b9d9b94a3a8f6c27e9e10db17f" candidate_uuid: "e0836be128054b62a424eedbe620f17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" is_pre_election: true
I20260620 20:09:26.464169 6059 raft_consensus.cc:2454] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0836be128054b62a424eedbe620f17c in term 0.
I20260620 20:09:26.464238 6193 raft_consensus.cc:2454] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0836be128054b62a424eedbe620f17c in term 0.
I20260620 20:09:26.464628 6266 leader_election.cc:302] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0836be128054b62a424eedbe620f17c, e0cabb282c3f4e16b086805651390732; no voters:
I20260620 20:09:26.464947 6408 raft_consensus.cc:2781] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:26.465077 6408 raft_consensus.cc:482] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:26.465173 6408 raft_consensus.cc:3037] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:26.467511 6408 raft_consensus.cc:504] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.468214 6408 leader_election.cc:288] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [CANDIDATE]: Term 1 election: Requested vote from peers e0cabb282c3f4e16b086805651390732 (127.4.169.129:41759), f39a81c01f1b439fa1ee8e08920d00fb (127.4.169.130:40019)
I20260620 20:09:26.468782 6193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5098b9d9b94a3a8f6c27e9e10db17f" candidate_uuid: "e0836be128054b62a424eedbe620f17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39a81c01f1b439fa1ee8e08920d00fb"
I20260620 20:09:26.468765 6059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5098b9d9b94a3a8f6c27e9e10db17f" candidate_uuid: "e0836be128054b62a424eedbe620f17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cabb282c3f4e16b086805651390732"
I20260620 20:09:26.469002 6193 raft_consensus.cc:3037] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:26.469017 6059 raft_consensus.cc:3037] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:26.472095 6059 raft_consensus.cc:2454] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0836be128054b62a424eedbe620f17c in term 1.
I20260620 20:09:26.472111 6193 raft_consensus.cc:2454] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0836be128054b62a424eedbe620f17c in term 1.
I20260620 20:09:26.472635 6263 leader_election.cc:302] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0836be128054b62a424eedbe620f17c, f39a81c01f1b439fa1ee8e08920d00fb; no voters:
I20260620 20:09:26.473011 6408 raft_consensus.cc:2781] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:26.473258 6408 raft_consensus.cc:686] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [term 1 LEADER]: Becoming Leader. State: Replica: e0836be128054b62a424eedbe620f17c, State: Running, Role: LEADER
I20260620 20:09:26.473594 6408 consensus_queue.cc:237] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } }
I20260620 20:09:26.475579 6420 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: TxnStatusTablet state changed. Reason: New leader e0836be128054b62a424eedbe620f17c. Latest consensus state: current_term: 1 leader_uuid: "e0836be128054b62a424eedbe620f17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.475965 6420 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:09:26.476714 6444 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:09:26.476984 6444 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:09:26.478152 5895 catalog_manager.cc:5495] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c reported cstate change: term changed from 0 to 1, leader changed from <none> to e0836be128054b62a424eedbe620f17c (127.4.169.131). New cstate: current_term: 1 leader_uuid: "e0836be128054b62a424eedbe620f17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } health_report { overall_health: HEALTHY } } }
I20260620 20:09:26.508186 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.500975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:45065 (local address 127.0.0.1:56308)
0620 20:09:26.501307 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:26.501331 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:26.501500 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:26.502250 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:26.502254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:26.502265 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:09:26.502523 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.502533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.504142 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.504148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.505305 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.505319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.506484 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.506490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.507062 (+ 572us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.507142 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:09:26.507697 (+ 555us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:09:26.507716 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:09:26.507798 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:09:26.507943 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:09:26.508287 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.501129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.4.169.131:45065)
0620 20:09:26.501846 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.501853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.501871 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:09:26.501946 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.501951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.502055 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.502153 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:26.502669 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.503948 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.505471 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.506219 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.506379 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.507382 (+ 1003us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:26.507557 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:26.507665 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.507862 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.507957 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573}
I20260620 20:09:26.512560 6059 raft_consensus.cc:1261] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Refusing update from remote peer e0836be128054b62a424eedbe620f17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:26.513363 6193 raft_consensus.cc:1261] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb [term 1 FOLLOWER]: Refusing update from remote peer e0836be128054b62a424eedbe620f17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:26.513348 6408 consensus_queue.cc:1035] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:26.513957 6408 consensus_queue.cc:1035] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:26.517063 6407 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: TxnStatusTablet state changed. Reason: New leader e0836be128054b62a424eedbe620f17c. Latest consensus state: current_term: 1 leader_uuid: "e0836be128054b62a424eedbe620f17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.517382 6407 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:26.519320 6411 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0836be128054b62a424eedbe620f17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.519683 6411 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:09:26.519914 6414 mvcc.cc:205] Tried to move back new op lower bound from 7299015338028908544 to 7299015337879216128. Current Snapshot: MvccSnapshot[applied={T|T < 7299015338028908544}]
I20260620 20:09:26.521584 6407 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0836be128054b62a424eedbe620f17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.521876 6407 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P f39a81c01f1b439fa1ee8e08920d00fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:26.525457 6409 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: TxnStatusTablet state changed. Reason: New leader e0836be128054b62a424eedbe620f17c. Latest consensus state: current_term: 1 leader_uuid: "e0836be128054b62a424eedbe620f17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.525754 6409 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:26.526154 6408 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0836be128054b62a424eedbe620f17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.526492 6408 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0836be128054b62a424eedbe620f17c: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:09:26.529111 6409 tablet_replica.cc:439] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0836be128054b62a424eedbe620f17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 } } peers { permanent_uuid: "f39a81c01f1b439fa1ee8e08920d00fb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40019 } } peers { permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 } } }
I20260620 20:09:26.529435 6409 tablet_replica.cc:442] T 5a5098b9d9b94a3a8f6c27e9e10db17f P e0cabb282c3f4e16b086805651390732: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:26.538849 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.532025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.4.169.130:40019)
0620 20:09:26.532829 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.532834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.532863 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:09:26.532946 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.532950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.533020 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.533200 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:26.533796 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.535050 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.536609 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.537032 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.537167 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.537614 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:26.538177 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:26.538350 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.538442 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.538521 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":509,"threads_started":1}
I20260620 20:09:26.555903 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.550109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:39665 (local address 127.4.169.131:45065)
0620 20:09:26.550329 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.550338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.550749 (+ 411us) server_negotiation.cc:407] Connection header received
0620 20:09:26.550985 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.550989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.551056 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.551162 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:26.551688 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.552491 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.553983 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.554526 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.554658 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.555099 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:26.555227 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:26.555332 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.555479 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.555560 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:09:26.555967 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.550001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:45065 (local address 127.4.169.130:39665)
0620 20:09:26.550643 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:26.550659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:26.550836 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:26.551177 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:26.551181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:26.551193 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:09:26.551539 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.551548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.552633 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.552639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.553843 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.553857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.554717 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.554724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.554848 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.554906 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:09:26.555344 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:09:26.555357 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 20:09:26.555457 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:09:26.555595 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":324,"threads_started":1}
I20260620 20:09:26.566819 6059 raft_consensus.cc:1261] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Refusing update from remote peer f39a81c01f1b439fa1ee8e08920d00fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:26.567135 6333 raft_consensus.cc:1261] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [term 1 FOLLOWER]: Refusing update from remote peer f39a81c01f1b439fa1ee8e08920d00fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:26.567516 6432 consensus_queue.cc:1035] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cabb282c3f4e16b086805651390732" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:26.567915 6407 consensus_queue.cc:1035] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0836be128054b62a424eedbe620f17c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:26.615279 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.608371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:43133 (local address 127.4.169.130:40019)
0620 20:09:26.608583 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.608593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.609186 (+ 593us) server_negotiation.cc:407] Connection header received
0620 20:09:26.609395 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.609400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.609488 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.609612 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:26.610194 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.611471 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.612939 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.613662 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.613823 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.614475 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:26.614625 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:26.614738 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.614860 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.614954 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:09:26.615464 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.608292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:40019 (local address 127.4.169.131:43133)
0620 20:09:26.609066 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:26.609093 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:26.609244 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:26.609627 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:26.609631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:26.609643 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:09:26.610012 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.610025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.611595 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.611599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.612794 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:26.612805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.613909 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.613916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:26.614115 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.614204 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:09:26.614740 (+ 536us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:09:26.614752 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 20:09:26.614841 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:09:26.615011 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":378,"threads_started":1}
I20260620 20:09:26.656833 5895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:26.657078 5895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:26.660161 5895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80054b47d6bc483c9a19beb573a7e8a2: Sending DeleteTablet for 3 replicas of tablet d2e21b09a24c4626ac3dc46de6c872b9
I20260620 20:09:26.661091 6034 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e21b09a24c4626ac3dc46de6c872b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:26 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:58018
I20260620 20:09:26.661978 6173 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e21b09a24c4626ac3dc46de6c872b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:26 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:46998
I20260620 20:09:26.672525 6312 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e21b09a24c4626ac3dc46de6c872b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:26 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:56296
I20260620 20:09:26.696300 6463 tablet_replica.cc:330] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: stopping tablet replica
I20260620 20:09:26.696732 6463 raft_consensus.cc:2229] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:26.697190 6463 raft_consensus.cc:2258] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:26.698930 6463 ts_tablet_manager.cc:1905] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:26.701819 6461 tablet_replica.cc:330] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: stopping tablet replica
I20260620 20:09:26.702240 6461 raft_consensus.cc:2229] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:09:26.702895 6460 tablet_replica.cc:330] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: stopping tablet replica
I20260620 20:09:26.703722 6461 raft_consensus.cc:2258] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:26.704119 6460 raft_consensus.cc:2229] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:26.704449 6460 raft_consensus.cc:2258] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:26.706095 6460 ts_tablet_manager.cc:1905] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:26.707800 6461 ts_tablet_manager.cc:1905] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:26.734349 6460 ts_tablet_manager.cc:1918] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:09:26.734659 6460 log.cc:1196] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/d2e21b09a24c4626ac3dc46de6c872b9
I20260620 20:09:26.735281 6460 ts_tablet_manager.cc:1939] T d2e21b09a24c4626ac3dc46de6c872b9 P e0cabb282c3f4e16b086805651390732: Deleting consensus metadata
I20260620 20:09:26.736756 5884 catalog_manager.cc:4841] TS e0cabb282c3f4e16b086805651390732 (127.4.169.129:41759): tablet d2e21b09a24c4626ac3dc46de6c872b9 (table test-table [id=45b590aa2eb144f18e5ee2dfb5d68b8e]) successfully deleted
I20260620 20:09:26.737769 6463 ts_tablet_manager.cc:1918] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:09:26.737939 6463 log.cc:1196] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/d2e21b09a24c4626ac3dc46de6c872b9
I20260620 20:09:26.738452 6463 ts_tablet_manager.cc:1939] T d2e21b09a24c4626ac3dc46de6c872b9 P e0836be128054b62a424eedbe620f17c: Deleting consensus metadata
I20260620 20:09:26.739665 5884 catalog_manager.cc:4841] TS e0836be128054b62a424eedbe620f17c (127.4.169.131:45065): tablet d2e21b09a24c4626ac3dc46de6c872b9 (table test-table [id=45b590aa2eb144f18e5ee2dfb5d68b8e]) successfully deleted
I20260620 20:09:26.740017 6461 ts_tablet_manager.cc:1918] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:09:26.740207 6461 log.cc:1196] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/d2e21b09a24c4626ac3dc46de6c872b9
I20260620 20:09:26.740737 6461 ts_tablet_manager.cc:1939] T d2e21b09a24c4626ac3dc46de6c872b9 P f39a81c01f1b439fa1ee8e08920d00fb: Deleting consensus metadata
I20260620 20:09:26.742467 5881 catalog_manager.cc:4841] TS f39a81c01f1b439fa1ee8e08920d00fb (127.4.169.130:40019): tablet d2e21b09a24c4626ac3dc46de6c872b9 (table test-table [id=45b590aa2eb144f18e5ee2dfb5d68b8e]) successfully deleted
Jun 20 20:09:26 dist-test-slave-4bwq krb5kdc[5852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:26.758745 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.734960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.4.169.129:41759)
0620 20:09:26.735157 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.735163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.737719 (+ 2556us) server_negotiation.cc:407] Connection header received
0620 20:09:26.738181 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.738189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.738288 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.738362 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:26.741286 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.742162 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.743730 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.744141 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.744294 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.748316 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:26.748337 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:26.748361 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:26.748421 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:26.755361 (+ 6940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.756257 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.756267 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.756271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.756435 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.756836 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.756841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.756845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.757258 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:26.757548 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.758137 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.758352 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 20:09:26.760257 6099 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58020
I20260620 20:09:26.813258 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.769952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.4.169.190:36267)
0620 20:09:26.770167 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.770173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.782888 (+ 12715us) server_negotiation.cc:407] Connection header received
0620 20:09:26.783096 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.783102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.783187 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.783260 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:26.799102 (+ 15842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.800260 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.802308 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.802992 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.803088 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.806351 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:26.806358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:26.806363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:26.806433 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:26.809974 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.810871 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.810877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.810881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.811037 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.811484 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.811489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.811493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.811859 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:26.812104 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.812677 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.812903 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 20:09:26.814955 5895 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40424
I20260620 20:09:26.817657 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 5968
I20260620 20:09:26.844485 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.819551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.4.169.129:41759)
0620 20:09:26.819727 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.819732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.826903 (+ 7171us) server_negotiation.cc:407] Connection header received
0620 20:09:26.827057 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.827063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.827153 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.827280 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:26.829014 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.830184 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.832096 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.832771 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.832882 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.836298 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:26.836306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:26.836311 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:26.836360 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:26.840552 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.841961 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.841968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.841972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.842128 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.842622 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.842629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.842632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.842992 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:26.843248 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.843848 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.844092 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 20:09:26.846122 6099 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036
W20260620 20:09:26.846797 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 5968: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:09:26.847012 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 5968
I20260620 20:09:26.872745 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6108
Jun 20 20:09:26 dist-test-slave-4bwq krb5kdc[5852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:09:26.893647 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.873577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.4.169.130:40019)
0620 20:09:26.873765 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.873775 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.873796 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:09:26.873968 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.873973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.874061 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.874248 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:26.876795 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.878021 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.879934 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.880654 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.880783 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.886119 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:26.886137 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:26.886158 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:26.886202 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:26.890458 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.891294 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.891304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.891308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.891500 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.891917 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.891922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.891925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.892380 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:26.892734 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.893044 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.893260 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 20:09:26.894320 6238 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47014
W20260620 20:09:26.895231 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6108: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:09:26.895396 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 6108
I20260620 20:09:26.914601 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6247
Jun 20 20:09:26 dist-test-slave-4bwq krb5kdc[5852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:09:26.933027 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:26.915480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.4.169.131:45065)
0620 20:09:26.915679 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:26.915684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:26.915766 (+ 82us) server_negotiation.cc:407] Connection header received
0620 20:09:26.915934 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:26.915938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:26.916001 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:26.916091 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:26.917721 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.918640 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.920429 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:26.920855 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:26.920952 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:26.926309 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:26.926326 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:26.926342 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:26.926460 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:26.929693 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.930848 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.930857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.930861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.931002 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:26.931417 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:26.931422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:26.931426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:26.931812 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:26.932159 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:26.932436 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 20:09:26.932675 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 20:09:26.933713 6377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56314
W20260620 20:09:26.934432 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6247: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:09:26.934613 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 6247
I20260620 20:09:26.957603 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 5865
I20260620 20:09:26.958757 5937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:27.077467 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 5865
2026-06-20T20:09:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5883 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:09:27 dist-test-slave-4bwq krb5kdc[6471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:09:27 dist-test-slave-4bwq krb5kdc[6471](info): set up 2 sockets
Jun 20 20:09:27 dist-test-slave-4bwq krb5kdc[6471](info): commencing operation
krb5kdc: starting...
W20260620 20:09:28.159245 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:09:28 dist-test-slave-4bwq krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:09:28.346882 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:40449
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:40449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:28.671821 6484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:28.672184 6484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:28.672338 6484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:28.679718 6484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:28.679891 6484 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:09:28.680011 6484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:28.680145 6484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:28.680250 6484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:28.680375 6484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:28.691167 6484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:40449
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:40449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:28.692514 6484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:28.693847 6484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:28.705859 6492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:28.705821 6490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:28.705901 6489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:28.707378 6484 server_base.cc:1034] running on GCE node
I20260620 20:09:28.708233 6484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:28.709709 6484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:28.710932 6484 hybrid_clock.cc:648] HybridClock initialized: now 1781986168710871 us; error 34 us; skew 500 ppm
I20260620 20:09:28.711318 6484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:28 dist-test-slave-4bwq krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986168, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:28.714634 6484 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:09:28.716189 6484 webserver.cc:460] Webserver started at http://127.4.169.190:44749/ using document root <none> and password file <none>
I20260620 20:09:28.716883 6484 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:28.716979 6484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:28.717238 6484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:28.720305 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "298a221c29594d04ae6252ceca4174a1"
format_stamp: "Formatted at 2026-06-20 20:09:28 on dist-test-slave-4bwq"
server_key: "c5d3d0259bf40c53fa1346a800ec6d11"
server_key_iv: "c1bc771d6ad4e94ef7997ac54ca5d344"
server_key_version: "encryptionkey@0"
I20260620 20:09:28.721074 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "298a221c29594d04ae6252ceca4174a1"
format_stamp: "Formatted at 2026-06-20 20:09:28 on dist-test-slave-4bwq"
server_key: "c5d3d0259bf40c53fa1346a800ec6d11"
server_key_iv: "c1bc771d6ad4e94ef7997ac54ca5d344"
server_key_version: "encryptionkey@0"
I20260620 20:09:28.726954 6484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:09:28.730835 6499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:28.732367 6484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:28.732548 6484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "298a221c29594d04ae6252ceca4174a1"
format_stamp: "Formatted at 2026-06-20 20:09:28 on dist-test-slave-4bwq"
server_key: "c5d3d0259bf40c53fa1346a800ec6d11"
server_key_iv: "c1bc771d6ad4e94ef7997ac54ca5d344"
server_key_version: "encryptionkey@0"
I20260620 20:09:28.732802 6484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:28.769718 6484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:28.773948 6484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:28.774221 6484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:28.799073 6484 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:40449
I20260620 20:09:28.799106 6560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:40449 every 8 connection(s)
I20260620 20:09:28.800508 6484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:28.804688 6561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:28.805212 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 6484
I20260620 20:09:28.805445 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:28.806260 4774 external_mini_cluster.cc:1428] Setting key eff9fa0fb1de2679d0396c822ac6473b
I20260620 20:09:28.819054 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1: Bootstrap starting.
Jun 20 20:09:28 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:28.822885 6561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:28.824277 6561 log.cc:824] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:28.827517 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1: No bootstrap required, opened a new log
I20260620 20:09:28.829787 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:28.809380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.4.169.190:40449)
0620 20:09:28.810263 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:28.810276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:28.810322 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:09:28.812072 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:28.812097 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:28.812550 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:28.812983 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:28.814535 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:28.815887 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:28.817113 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:28.817815 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:28.818020 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:28.822241 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:28.822250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:28.822267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:28.822308 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:28.825577 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:28.826320 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:28.826328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:28.826334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:28.826579 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:28.826973 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:28.826982 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:28.826984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:28.827537 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:28.827827 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:28.828292 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 20:09:28.828722 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":665,"thread_start_us":413,"threads_started":1}
I20260620 20:09:28.838596 6561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298a221c29594d04ae6252ceca4174a1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40449 } }
I20260620 20:09:28.839071 6561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:28.839191 6561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298a221c29594d04ae6252ceca4174a1, State: Initialized, Role: FOLLOWER
I20260620 20:09:28.839782 6561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298a221c29594d04ae6252ceca4174a1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40449 } }
I20260620 20:09:28.840066 6561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:28.840217 6561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:28.840395 6561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:28.917832 6561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298a221c29594d04ae6252ceca4174a1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40449 } }
I20260620 20:09:28.918658 6561 leader_election.cc:302] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 298a221c29594d04ae6252ceca4174a1; no voters:
I20260620 20:09:28.919469 6561 leader_election.cc:288] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:28.919646 6566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:28.921056 6566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [term 1 LEADER]: Becoming Leader. State: Replica: 298a221c29594d04ae6252ceca4174a1, State: Running, Role: LEADER
I20260620 20:09:28.921615 6566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298a221c29594d04ae6252ceca4174a1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40449 } }
I20260620 20:09:28.921918 6561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:28.924523 6567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "298a221c29594d04ae6252ceca4174a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298a221c29594d04ae6252ceca4174a1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40449 } } }
I20260620 20:09:28.924767 6568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298a221c29594d04ae6252ceca4174a1. Latest consensus state: current_term: 1 leader_uuid: "298a221c29594d04ae6252ceca4174a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298a221c29594d04ae6252ceca4174a1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40449 } } }
I20260620 20:09:28.925098 6567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:28.925108 6568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:28.927394 6571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:28.931913 6571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:28.941299 6571 catalog_manager.cc:1347] Generated new cluster ID: 8ee4b9938537482f9196be70aa34c4ce
I20260620 20:09:28.941547 6571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:28.959776 6571 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:28.961202 6571 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:28.969054 6571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1: Generated new TSK 0
I20260620 20:09:28.970001 6571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
W20260620 20:09:30.029762 4774 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.169.129: real 1.053s user 0.001s sys 0.011s
I20260620 20:09:30.030866 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40449
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:30.358407 6587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:30.358738 6587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:30.358932 6587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:30.366453 6587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:30.366580 6587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:30.366751 6587 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:09:30.366838 6587 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:09:30.366961 6587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:09:30.377138 6587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.169.190:40449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:30.378561 6587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:30.379926 6587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:30.393707 6592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:30.393757 6593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:30.393671 6595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:30.394624 6587 server_base.cc:1034] running on GCE node
I20260620 20:09:30.395233 6587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:30.396168 6587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:30.397404 6587 hybrid_clock.cc:648] HybridClock initialized: now 1781986170397369 us; error 53 us; skew 500 ppm
I20260620 20:09:30.397732 6587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:30 dist-test-slave-4bwq krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986170, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:30.404366 6587 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:09:30.406045 6587 webserver.cc:460] Webserver started at http://127.4.169.129:42701/ using document root <none> and password file <none>
I20260620 20:09:30.406863 6587 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:30.406971 6587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:30.407246 6587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:30.410179 6587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "ee4174d7885645e896ca30bda3eec81d"
format_stamp: "Formatted at 2026-06-20 20:09:30 on dist-test-slave-4bwq"
server_key: "f8ab4438cd9fa0d792c77fe18eadd30f"
server_key_iv: "726c9db888bc6c39e4d3734ee8ea54cb"
server_key_version: "encryptionkey@0"
I20260620 20:09:30.411018 6587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "ee4174d7885645e896ca30bda3eec81d"
format_stamp: "Formatted at 2026-06-20 20:09:30 on dist-test-slave-4bwq"
server_key: "f8ab4438cd9fa0d792c77fe18eadd30f"
server_key_iv: "726c9db888bc6c39e4d3734ee8ea54cb"
server_key_version: "encryptionkey@0"
I20260620 20:09:30.416420 6587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:09:30.419911 6602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:30.421411 6587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:09:30.421599 6587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "ee4174d7885645e896ca30bda3eec81d"
format_stamp: "Formatted at 2026-06-20 20:09:30 on dist-test-slave-4bwq"
server_key: "f8ab4438cd9fa0d792c77fe18eadd30f"
server_key_iv: "726c9db888bc6c39e4d3734ee8ea54cb"
server_key_version: "encryptionkey@0"
I20260620 20:09:30.421824 6587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:30.437721 6587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:30.441953 6587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:30.442246 6587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:30.445133 6587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:30.445271 6587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:30.445444 6587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:30.445538 6587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:09:30 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986170, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:30.471369 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:30.445544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:47441 (local address 127.4.169.190:40449)
0620 20:09:30.446083 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:30.446092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:30.447281 (+ 1189us) server_negotiation.cc:407] Connection header received
0620 20:09:30.449464 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:30.449470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:30.450168 (+ 698us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:30.450308 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:30.453023 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:30.454034 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:30.456628 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:30.457262 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:30.457437 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:30.463292 (+ 5855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:30.463302 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:30.463308 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:30.463363 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:30.467125 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:30.468378 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:30.468386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:30.468390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:30.468574 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:30.469309 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:30.469315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:30.469319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:30.469611 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:30.469793 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:30.470619 (+ 826us) server_negotiation.cc:299] Negotiation successful
0620 20:09:30.470856 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":306,"threads_started":1}
I20260620 20:09:30.472568 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:30.446255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40449 (local address 127.4.169.129:47441)
0620 20:09:30.447054 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:30.447100 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:30.449170 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:30.450883 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:30.450892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:30.451819 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:30.452753 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:30.452773 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:30.454309 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:30.454317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:30.456450 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:30.456466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:30.457657 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:30.457670 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:30.458092 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:30.459140 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:30.459183 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:30.462989 (+ 3806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:30.467411 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:30.467422 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:30.467446 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:30.468195 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:30.468737 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:30.468745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:30.468754 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:30.469176 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:30.469765 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:30.469774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:30.470181 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 20:09:30.470508 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 20:09:30.471123 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":318,"threads_started":1}
I20260620 20:09:30.497665 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:30.479333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:57331 (local address 127.4.169.190:40449)
0620 20:09:30.479541 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:30.479550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:30.480247 (+ 697us) server_negotiation.cc:407] Connection header received
0620 20:09:30.480504 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:30.480508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:30.480575 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:30.480679 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:30.483071 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:30.483841 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:30.486740 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:30.487224 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:30.487348 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:30.490507 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:30.490515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:30.490523 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:30.490590 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:30.494304 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:30.495413 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:30.495421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:30.495426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:30.495643 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:30.496114 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:30.496119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:30.496124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:30.496469 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:30.496624 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:30.497042 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 20:09:30.497283 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:09:30.497799 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:30.479327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40449 (local address 127.4.169.129:57331)
0620 20:09:30.480111 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:30.480139 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:30.480335 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:30.480731 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:30.480739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:30.481546 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:30.482078 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:30.482103 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:30.484113 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:30.484119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:30.486548 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:30.486573 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:30.487478 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:30.487487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:30.487739 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:30.488571 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:30.488603 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:30.490221 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:30.494517 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:30.494529 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:30.494535 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:30.495255 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:30.495819 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:30.495824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:30.495828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:30.495983 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:30.496589 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:30.496594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:30.496785 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 20:09:30.496939 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 20:09:30.497303 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":372,"threads_started":1}
I20260620 20:09:30.508325 6587 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:33775
I20260620 20:09:30.508363 6722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:33775 every 8 connection(s)
I20260620 20:09:30.509603 6587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:09:30.510234 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 6587
I20260620 20:09:30.510499 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:09:30.511281 4774 external_mini_cluster.cc:1428] Setting key d2816e12e7b58afdb8ed55cba487f925
I20260620 20:09:30.512140 6723 heartbeater.cc:347] Connected to a master server at 127.4.169.190:40449
I20260620 20:09:30.512442 6723 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:30.513453 6723 heartbeater.cc:511] Master 127.4.169.190:40449 requested a full tablet report, sending...
I20260620 20:09:30.515900 6516 ts_manager.cc:194] Registered new tserver with Master: ee4174d7885645e896ca30bda3eec81d (127.4.169.129:33775)
I20260620 20:09:30.517375 6516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:47441
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
I20260620 20:09:31.520290 6723 heartbeater.cc:503] Master 127.4.169.190:40449 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
W20260620 20:09:31.603289 4774 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.169.130: real 1.090s user 0.004s sys 0.005s
I20260620 20:09:31.604599 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40449
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:31.935772 6727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:31.936180 6727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:31.936371 6727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:31.944140 6727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:31.944259 6727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:31.944409 6727 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:09:31.944491 6727 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:09:31.944625 6727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:09:31.954704 6727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.169.190:40449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:31.956029 6727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:31.957362 6727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:31.970938 6733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:31.970916 6735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:31.970911 6732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:31.976025 6727 server_base.cc:1034] running on GCE node
I20260620 20:09:31.976716 6727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:31.977722 6727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:31.978940 6727 hybrid_clock.cc:648] HybridClock initialized: now 1781986171978879 us; error 65 us; skew 500 ppm
I20260620 20:09:31.979260 6727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:31 dist-test-slave-4bwq krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986171, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:31.982455 6727 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:09:31.984066 6727 webserver.cc:460] Webserver started at http://127.4.169.130:39213/ using document root <none> and password file <none>
I20260620 20:09:31.984813 6727 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:31.984959 6727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:31.985271 6727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:31.988269 6727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "a8d27450e651486bbf8db3ef6327a390"
format_stamp: "Formatted at 2026-06-20 20:09:31 on dist-test-slave-4bwq"
server_key: "28a5f2b7af77399969d8e86c710643ef"
server_key_iv: "f6be2e36677395f0e676b2a331c87f17"
server_key_version: "encryptionkey@0"
I20260620 20:09:31.989048 6727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "a8d27450e651486bbf8db3ef6327a390"
format_stamp: "Formatted at 2026-06-20 20:09:31 on dist-test-slave-4bwq"
server_key: "28a5f2b7af77399969d8e86c710643ef"
server_key_iv: "f6be2e36677395f0e676b2a331c87f17"
server_key_version: "encryptionkey@0"
I20260620 20:09:31.994648 6727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 20:09:31.998405 6742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:32.000031 6727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:32.000262 6727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "a8d27450e651486bbf8db3ef6327a390"
format_stamp: "Formatted at 2026-06-20 20:09:31 on dist-test-slave-4bwq"
server_key: "28a5f2b7af77399969d8e86c710643ef"
server_key_iv: "f6be2e36677395f0e676b2a331c87f17"
server_key_version: "encryptionkey@0"
I20260620 20:09:32.000505 6727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:32.022804 6727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:32.027189 6727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:32.027509 6727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:32.030195 6727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:32.030354 6727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:32.030483 6727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:32.030571 6727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:09:32 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986171, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:32.056967 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.030898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:37729 (local address 127.4.169.190:40449)
0620 20:09:32.031377 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.031386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.032795 (+ 1409us) server_negotiation.cc:407] Connection header received
0620 20:09:32.035078 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.035085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.035178 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.035321 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:32.037890 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.039294 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.041709 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.042515 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.042673 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.048469 (+ 5796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:32.048479 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:32.048487 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:32.048534 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:32.053114 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.054426 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.054434 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.054438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.054589 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.055095 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.055101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.055104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.055393 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:32.055545 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.056328 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.056553 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":230,"threads_started":1}
I20260620 20:09:32.058741 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.031719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40449 (local address 127.4.169.130:37729)
0620 20:09:32.032621 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.032669 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.034661 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.035727 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.035742 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.036788 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:32.037646 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.037700 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.039487 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.039507 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.041580 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.041595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.043084 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.043095 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.043548 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.044417 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:32.044464 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:32.048164 (+ 3700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:32.053277 (+ 5113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.053288 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.053308 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.054231 (+ 923us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.054754 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.054758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.054760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.054954 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.055534 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:32.055543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:32.055947 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.056254 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.056830 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":370,"threads_started":1}
I20260620 20:09:32.078580 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.065277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:35781 (local address 127.4.169.190:40449)
0620 20:09:32.065497 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.065505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.066317 (+ 812us) server_negotiation.cc:407] Connection header received
0620 20:09:32.066582 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.066586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.066645 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.066747 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:32.067731 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.068464 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.069828 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.070251 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.070448 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.072570 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:32.072579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:32.072586 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:32.072633 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:32.075640 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.076460 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.076466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.076470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.076617 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.077015 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.077021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.077025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.077369 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:32.077525 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.077949 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.078202 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:09:32.078716 6809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.065238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40449 (local address 127.4.169.130:35781)
0620 20:09:32.066208 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.066230 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.066435 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.066779 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.066783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.067273 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:32.067595 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.067603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.068633 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.068639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.069697 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.069709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.070526 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.070532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.070727 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.071567 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:32.071584 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:32.072368 (+ 784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:32.075791 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.075798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.075802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.076335 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.076761 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.076766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.076770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.076907 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.077490 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:32.077495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:32.077693 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.077846 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.078191 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":618,"threads_started":1}
I20260620 20:09:32.094365 6727 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:38989
I20260620 20:09:32.094494 6862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:38989 every 8 connection(s)
I20260620 20:09:32.095706 6727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:09:32.097939 6863 heartbeater.cc:347] Connected to a master server at 127.4.169.190:40449
I20260620 20:09:32.098177 6863 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:32.098927 6863 heartbeater.cc:511] Master 127.4.169.190:40449 requested a full tablet report, sending...
I20260620 20:09:32.100297 6516 ts_manager.cc:194] Registered new tserver with Master: a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989)
I20260620 20:09:32.101151 6516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:37729
I20260620 20:09:32.103900 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 6727
I20260620 20:09:32.104100 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:09:32.104614 4774 external_mini_cluster.cc:1428] Setting key 028fd89d855d13b343f2c2465b2c69c5
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:09:32.159394 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40449
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:32.480495 6867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:32.480822 6867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:32.480994 6867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:32.488416 6867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:32.488585 6867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:32.488749 6867 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:09:32.488839 6867 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:09:32.488979 6867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:09:32.498883 6867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.169.190:40449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:32.500252 6867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:32.501650 6867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:32.516970 6872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:32.516896 6873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:32.516944 6875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:32.517773 6867 server_base.cc:1034] running on GCE node
I20260620 20:09:32.518597 6867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:32.519685 6867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:32.520974 6867 hybrid_clock.cc:648] HybridClock initialized: now 1781986172520894 us; error 64 us; skew 500 ppm
I20260620 20:09:32.521360 6867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:32 dist-test-slave-4bwq krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986172, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:32.524683 6867 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:09:32.526261 6867 webserver.cc:460] Webserver started at http://127.4.169.131:46617/ using document root <none> and password file <none>
I20260620 20:09:32.527022 6867 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:32.527122 6867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:32.527396 6867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:32.530144 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "dc8ebb583a1c4a598cd3ac04b6475e71"
format_stamp: "Formatted at 2026-06-20 20:09:32 on dist-test-slave-4bwq"
server_key: "1dc27e4fafc545f95154d3621c2a6eff"
server_key_iv: "7bf179862bc640db27545b9c649f4f1d"
server_key_version: "encryptionkey@0"
I20260620 20:09:32.530931 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "dc8ebb583a1c4a598cd3ac04b6475e71"
format_stamp: "Formatted at 2026-06-20 20:09:32 on dist-test-slave-4bwq"
server_key: "1dc27e4fafc545f95154d3621c2a6eff"
server_key_iv: "7bf179862bc640db27545b9c649f4f1d"
server_key_version: "encryptionkey@0"
I20260620 20:09:32.535934 6867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:09:32.539810 6882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:32.541127 6867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:32.541316 6867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "dc8ebb583a1c4a598cd3ac04b6475e71"
format_stamp: "Formatted at 2026-06-20 20:09:32 on dist-test-slave-4bwq"
server_key: "1dc27e4fafc545f95154d3621c2a6eff"
server_key_iv: "7bf179862bc640db27545b9c649f4f1d"
server_key_version: "encryptionkey@0"
I20260620 20:09:32.541536 6867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:32.573849 6867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:32.578272 6867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:32.578598 6867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:32.581410 6867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:32.581529 6867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:32.581645 6867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:32.581773 6867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:09:32 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986172, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:32.604557 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.581950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:41151 (local address 127.4.169.190:40449)
0620 20:09:32.582797 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.582801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.583540 (+ 739us) server_negotiation.cc:407] Connection header received
0620 20:09:32.585793 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.585799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.585892 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.586011 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:32.588262 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.589564 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.592174 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.592606 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.592746 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.597616 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:32.597621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:32.597624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:32.597680 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:32.600944 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.601889 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.601895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.601899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.602057 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.602667 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.602674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.602677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.603068 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:32.603231 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.603942 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.604149 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":670,"threads_started":1}
I20260620 20:09:32.605664 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.582607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40449 (local address 127.4.169.131:41151)
0620 20:09:32.583385 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.583426 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.585474 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.586280 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.586296 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.587206 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:32.588039 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.588081 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.589744 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.589748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.592019 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.592030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.592857 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.592865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.593076 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.593813 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:32.593850 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:32.597371 (+ 3521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:32.601085 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.601095 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.601120 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.601741 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.602217 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.602223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.602226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.602540 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.603201 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:32.603208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:32.603588 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.603870 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.604401 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":348,"threads_started":1}
I20260620 20:09:32.627010 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.611541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:47467 (local address 127.4.169.190:40449)
0620 20:09:32.611785 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.611794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.612401 (+ 607us) server_negotiation.cc:407] Connection header received
0620 20:09:32.612540 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.612549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.612664 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.612806 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:32.613628 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.615122 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.616339 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.617206 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.617358 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.620377 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:32.620385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:32.620393 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:32.620439 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:32.623617 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.624669 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.624674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.624676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.624798 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.625314 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.625321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.625326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.625685 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:32.625805 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.626492 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.626772 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:09:32.627534 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.611664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40449 (local address 127.4.169.131:47467)
0620 20:09:32.612304 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.612326 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.612426 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.612815 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.612818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.613206 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:32.613497 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.613505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.615252 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.615255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.616206 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.616217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.617457 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.617462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.617594 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.618376 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:32.618441 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:32.620064 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:32.623765 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.623774 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.623781 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.624530 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.624922 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.624930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.624936 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.625147 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.625815 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:32.625824 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:32.626123 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.626318 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.626846 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":352,"threads_started":1}
I20260620 20:09:32.639390 6867 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:37031
I20260620 20:09:32.639439 7002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:37031 every 8 connection(s)
I20260620 20:09:32.640686 6867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:09:32.642832 7003 heartbeater.cc:347] Connected to a master server at 127.4.169.190:40449
I20260620 20:09:32.643069 7003 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:32.643713 7003 heartbeater.cc:511] Master 127.4.169.190:40449 requested a full tablet report, sending...
I20260620 20:09:32.645123 6516 ts_manager.cc:194] Registered new tserver with Master: dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031)
I20260620 20:09:32.645910 6516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:41151
I20260620 20:09:32.648231 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 6867
I20260620 20:09:32.648427 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:09:32.648938 4774 external_mini_cluster.cc:1428] Setting key 37e8546585ef6fd37b7ef948360044d5
I20260620 20:09:32.652057 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:09:32 dist-test-slave-4bwq krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:09:32 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:32.699481 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.685966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.4.169.190:40449)
0620 20:09:32.686204 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.686211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.686359 (+ 148us) server_negotiation.cc:407] Connection header received
0620 20:09:32.686605 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.686610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.686674 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.686776 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:32.688646 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.689365 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.690586 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.690988 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.691081 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.694768 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:32.694772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:32.694775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:32.694808 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:32.697145 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.697741 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.697744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.697746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.697873 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.698173 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.698176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.698178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.698491 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:32.698665 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.698944 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.699104 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 20:09:32.703558 6516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:09:32.710911 6516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:32.749214 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.723215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37031 (local address 127.0.0.1:47176)
0620 20:09:32.737857 (+ 14642us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.737900 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.739043 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.740585 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.740597 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.740647 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:32.741155 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.741171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.743928 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.743938 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.746304 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.746318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.747828 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.747841 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.748363 (+ 522us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.748425 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.748588 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.748804 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14373,"thread_start_us":15551,"threads_started":1}
I20260620 20:09:32.749949 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.724000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.4.169.131:37031)
0620 20:09:32.724886 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.724898 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.738051 (+ 13153us) server_negotiation.cc:407] Connection header received
0620 20:09:32.739401 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.739415 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.739797 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.740095 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:32.741445 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.743730 (+ 2285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.746489 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.747613 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.747785 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.748047 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.749460 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.749621 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":776,"threads_started":1}
I20260620 20:09:32.751324 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.723348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:33775 (local address 127.0.0.1:59188)
0620 20:09:32.739060 (+ 15712us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.739079 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.739197 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.744322 (+ 5125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.744326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.744355 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:32.744851 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.744864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.746712 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.746717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.749028 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.749046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.750558 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.750564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.750737 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.750776 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.750857 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.750979 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15583,"mutex_wait_us":65,"thread_start_us":14084,"threads_started":1}
I20260620 20:09:32.751377 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.724605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.4.169.129:33775)
0620 20:09:32.743652 (+ 19047us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.743663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.743697 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:09:32.743812 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.743820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.744100 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.745034 (+ 934us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:32.745237 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.746582 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.749233 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.750312 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.750505 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.750704 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.750928 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.751095 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18871,"thread_start_us":18680,"threads_started":1}
I20260620 20:09:32.754009 6933 tablet_service.cc:1467] Processing CreateTablet for tablet b135985df1524cf98a8fc8a9a1bdcd28 (DEFAULT_TABLE table=test-table [id=8dcfb6631d1e4c55ae11efb6412f5ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:32.754942 6653 tablet_service.cc:1467] Processing CreateTablet for tablet b135985df1524cf98a8fc8a9a1bdcd28 (DEFAULT_TABLE table=test-table [id=8dcfb6631d1e4c55ae11efb6412f5ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:32.755489 6933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b135985df1524cf98a8fc8a9a1bdcd28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:32.756062 6653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b135985df1524cf98a8fc8a9a1bdcd28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:32.760933 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.727447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:38989 (local address 127.0.0.1:36296)
0620 20:09:32.743947 (+ 16500us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.743965 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.744098 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.745273 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.745277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.745309 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:32.745700 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.745711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.747575 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.747580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.750674 (+ 3094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.750687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.760002 (+ 9315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.760012 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.760211 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.760245 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.760339 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.760529 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16363,"mutex_wait_us":100,"thread_start_us":16206,"threads_started":1}
I20260620 20:09:32.760993 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.727729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.4.169.130:38989)
0620 20:09:32.742877 (+ 15148us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.742890 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.744270 (+ 1380us) server_negotiation.cc:407] Connection header received
0620 20:09:32.744381 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.744392 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.744664 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.745024 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:32.745917 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.747416 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.750802 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.759469 (+ 8667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.759627 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.759841 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.760544 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.760723 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14906,"thread_start_us":15163,"threads_started":1}
I20260620 20:09:32.765079 6793 tablet_service.cc:1467] Processing CreateTablet for tablet b135985df1524cf98a8fc8a9a1bdcd28 (DEFAULT_TABLE table=test-table [id=8dcfb6631d1e4c55ae11efb6412f5ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:32.766551 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b135985df1524cf98a8fc8a9a1bdcd28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:32.768473 7022 tablet_bootstrap.cc:492] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: Bootstrap starting.
I20260620 20:09:32.771142 7023 tablet_bootstrap.cc:492] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: Bootstrap starting.
I20260620 20:09:32.771737 7022 tablet_bootstrap.cc:654] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:32.773047 7022 log.cc:824] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:32.774526 7023 tablet_bootstrap.cc:654] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:32.775728 7023 log.cc:824] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:32.776404 7022 tablet_bootstrap.cc:492] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: No bootstrap required, opened a new log
I20260620 20:09:32.776796 7022 ts_tablet_manager.cc:1397] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 20:09:32.780594 7026 tablet_bootstrap.cc:492] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: Bootstrap starting.
I20260620 20:09:32.783480 7023 tablet_bootstrap.cc:492] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: No bootstrap required, opened a new log
I20260620 20:09:32.783806 7023 ts_tablet_manager.cc:1397] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20260620 20:09:32.784025 7026 tablet_bootstrap.cc:654] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:32.785050 7026 log.cc:824] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:32.788317 7026 tablet_bootstrap.cc:492] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: No bootstrap required, opened a new log
I20260620 20:09:32.788584 7026 ts_tablet_manager.cc:1397] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 20:09:32.788970 7022 raft_consensus.cc:348] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } }
I20260620 20:09:32.789407 7022 raft_consensus.cc:374] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:32.789542 7022 raft_consensus.cc:729] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4174d7885645e896ca30bda3eec81d, State: Initialized, Role: FOLLOWER
I20260620 20:09:32.790177 7022 consensus_queue.cc:260] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } }
I20260620 20:09:32.792347 7022 ts_tablet_manager.cc:1428] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260620 20:09:32.793671 7023 raft_consensus.cc:348] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } }
I20260620 20:09:32.794044 7023 raft_consensus.cc:374] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:32.794167 7023 raft_consensus.cc:729] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8ebb583a1c4a598cd3ac04b6475e71, State: Initialized, Role: FOLLOWER
I20260620 20:09:32.794763 7023 consensus_queue.cc:260] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } }
I20260620 20:09:32.795836 7003 heartbeater.cc:503] Master 127.4.169.190:40449 was elected leader, sending a full tablet report...
I20260620 20:09:32.796511 7023 ts_tablet_manager.cc:1428] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 20:09:32.799031 7026 raft_consensus.cc:348] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } }
I20260620 20:09:32.799523 7026 raft_consensus.cc:374] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:32.799636 7026 raft_consensus.cc:729] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d27450e651486bbf8db3ef6327a390, State: Initialized, Role: FOLLOWER
I20260620 20:09:32.800225 7026 consensus_queue.cc:260] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } }
I20260620 20:09:32.801187 6863 heartbeater.cc:503] Master 127.4.169.190:40449 was elected leader, sending a full tablet report...
I20260620 20:09:32.801815 7026 ts_tablet_manager.cc:1428] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 20:09:32.834928 7028 raft_consensus.cc:482] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:32.835242 7028 raft_consensus.cc:504] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } }
I20260620 20:09:32.837157 7028 leader_election.cc:288] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031), a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989)
I20260620 20:09:32.844444 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.837641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:54301 (local address 127.4.169.131:37031)
0620 20:09:32.837837 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.837845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.838858 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 20:09:32.839217 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.839225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.839322 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.839419 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:32.839978 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.840689 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.842769 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.843513 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.843701 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.843776 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.844043 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.844164 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:09:32.844548 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.837383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37031 (local address 127.4.169.129:54301)
0620 20:09:32.838741 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.838767 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.839012 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.839432 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.839436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.839458 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:32.839836 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.839844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.840819 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.840827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.842623 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.842634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.843723 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.843729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.843878 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.843901 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.844019 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.844150 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"mutex_wait_us":72,"thread_start_us":934,"threads_started":1}
I20260620 20:09:32.844872 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.837688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:38989 (local address 127.4.169.129:42055)
0620 20:09:32.838979 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.838996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.839138 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.839526 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.839530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.839568 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:32.839969 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.839977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.841007 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.841016 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.842947 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.842958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.844057 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.844063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.844211 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.844232 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.844339 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.844462 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":917,"threads_started":1}
I20260620 20:09:32.845439 6958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b135985df1524cf98a8fc8a9a1bdcd28" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" is_pre_election: true
I20260620 20:09:32.845966 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.838035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:42055 (local address 127.4.169.130:38989)
0620 20:09:32.838451 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.838457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.839083 (+ 626us) server_negotiation.cc:407] Connection header received
0620 20:09:32.839339 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.839343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.839401 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.839507 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:32.840113 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.840850 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.843088 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.843834 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.844020 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.844094 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.844363 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.844487 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260620 20:09:32.845983 6958 raft_consensus.cc:2454] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 0.
I20260620 20:09:32.846688 6603 leader_election.cc:302] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc8ebb583a1c4a598cd3ac04b6475e71, ee4174d7885645e896ca30bda3eec81d; no voters:
I20260620 20:09:32.846942 6818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b135985df1524cf98a8fc8a9a1bdcd28" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d27450e651486bbf8db3ef6327a390" is_pre_election: true
I20260620 20:09:32.847067 7028 raft_consensus.cc:2781] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:32.847223 7028 raft_consensus.cc:482] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:32.847406 7028 raft_consensus.cc:3037] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:32.847519 6818 raft_consensus.cc:2454] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 0.
W20260620 20:09:32.847747 6864 tablet.cc:2369] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:32.849927 7028 raft_consensus.cc:504] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } }
I20260620 20:09:32.850564 7028 leader_election.cc:288] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 election: Requested vote from peers dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031), a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989)
I20260620 20:09:32.850869 6958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b135985df1524cf98a8fc8a9a1bdcd28" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71"
I20260620 20:09:32.851046 6958 raft_consensus.cc:3037] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:32.851056 6818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b135985df1524cf98a8fc8a9a1bdcd28" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d27450e651486bbf8db3ef6327a390"
I20260620 20:09:32.851300 6818 raft_consensus.cc:3037] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:32.853161 6958 raft_consensus.cc:2454] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 1.
I20260620 20:09:32.853437 6818 raft_consensus.cc:2454] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 1.
I20260620 20:09:32.853499 6603 leader_election.cc:302] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc8ebb583a1c4a598cd3ac04b6475e71, ee4174d7885645e896ca30bda3eec81d; no voters:
I20260620 20:09:32.853791 7028 raft_consensus.cc:2781] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:32.854194 7028 raft_consensus.cc:686] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 1 LEADER]: Becoming Leader. State: Replica: ee4174d7885645e896ca30bda3eec81d, State: Running, Role: LEADER
I20260620 20:09:32.854707 7028 consensus_queue.cc:237] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } }
I20260620 20:09:32.859027 6515 catalog_manager.cc:5495] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee4174d7885645e896ca30bda3eec81d (127.4.169.129). New cstate: current_term: 1 leader_uuid: "ee4174d7885645e896ca30bda3eec81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } health_report { overall_health: UNKNOWN } } }
W20260620 20:09:32.892663 7004 tablet.cc:2369] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:09:32 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986168, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:32.905779 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.892987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40449 (local address 127.0.0.1:55972)
0620 20:09:32.893576 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.893592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.893758 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.894080 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.894084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.894825 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:32.895101 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.895111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.896096 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.896103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.897033 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.897044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.897803 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.897816 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.897953 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.898615 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:32.898635 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:32.900741 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:32.903365 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.903378 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.903386 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.903766 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.904132 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:32.904139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:32.904143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:32.904277 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:32.904820 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:32.904827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:32.905066 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.905221 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.905456 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":341,"threads_started":1}
I20260620 20:09:32.905812 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.892918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.4.169.190:40449)
0620 20:09:32.893090 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.893094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.893680 (+ 586us) server_negotiation.cc:407] Connection header received
0620 20:09:32.893884 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.893889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.893953 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.894055 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:32.895237 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.895970 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.897184 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.897628 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.897760 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.900924 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:32.900929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:32.900932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:32.900957 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:32.903237 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.903898 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.903905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.903911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.904030 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:32.904375 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:32.904383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:32.904387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:32.904641 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:32.904814 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.905303 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.905569 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":52}
I20260620 20:09:32.911208 6515 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:55972:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:09:32.912315 6515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:32.919183 6793 tablet_service.cc:1467] Processing CreateTablet for tablet aa5d185cbd9b4118ba3e233d7871e916 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6686a7f4e08e480fa6b181c11291806e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:09:32.919359 6933 tablet_service.cc:1467] Processing CreateTablet for tablet aa5d185cbd9b4118ba3e233d7871e916 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6686a7f4e08e480fa6b181c11291806e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:09:32.919114 6653 tablet_service.cc:1467] Processing CreateTablet for tablet aa5d185cbd9b4118ba3e233d7871e916 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6686a7f4e08e480fa6b181c11291806e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:09:32.920179 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5d185cbd9b4118ba3e233d7871e916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:32.920188 6933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5d185cbd9b4118ba3e233d7871e916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:32.920188 6653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5d185cbd9b4118ba3e233d7871e916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:32.926367 7026 tablet_bootstrap.cc:492] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: Bootstrap starting.
I20260620 20:09:32.927899 7023 tablet_bootstrap.cc:492] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: Bootstrap starting.
I20260620 20:09:32.927989 7026 tablet_bootstrap.cc:654] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:32.928570 7022 tablet_bootstrap.cc:492] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: Bootstrap starting.
I20260620 20:09:32.929559 7023 tablet_bootstrap.cc:654] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:32.930022 7022 tablet_bootstrap.cc:654] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:32.930449 7026 tablet_bootstrap.cc:492] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: No bootstrap required, opened a new log
I20260620 20:09:32.930645 7026 ts_tablet_manager.cc:1397] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260620 20:09:32.931516 7023 tablet_bootstrap.cc:492] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: No bootstrap required, opened a new log
I20260620 20:09:32.931663 7023 ts_tablet_manager.cc:1397] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 20:09:32.931583 7026 raft_consensus.cc:348] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:32.931895 7026 raft_consensus.cc:374] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:32.932016 7026 raft_consensus.cc:729] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d27450e651486bbf8db3ef6327a390, State: Initialized, Role: FOLLOWER
I20260620 20:09:32.932346 7026 consensus_queue.cc:260] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:32.932440 7023 raft_consensus.cc:348] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:32.932672 7023 raft_consensus.cc:374] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:32.932763 7023 raft_consensus.cc:729] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8ebb583a1c4a598cd3ac04b6475e71, State: Initialized, Role: FOLLOWER
I20260620 20:09:32.933007 7023 consensus_queue.cc:260] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:32.933171 7026 ts_tablet_manager.cc:1428] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 20:09:32.932925 7030 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:32.933332 7030 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:32.933879 7029 raft_consensus.cc:482] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:32.934048 7029 raft_consensus.cc:504] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:32.935892 7029 leader_election.cc:288] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee4174d7885645e896ca30bda3eec81d (127.4.169.129:33775), a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989)
I20260620 20:09:32.936173 7029 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:32.936439 7029 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:32.938287 7023 ts_tablet_manager.cc:1428] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260620 20:09:32.944105 7022 tablet_bootstrap.cc:492] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: No bootstrap required, opened a new log
I20260620 20:09:32.944278 7022 ts_tablet_manager.cc:1397] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260620 20:09:32.945098 7022 raft_consensus.cc:348] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:32.945325 7022 raft_consensus.cc:374] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:32.945426 7022 raft_consensus.cc:729] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4174d7885645e896ca30bda3eec81d, State: Initialized, Role: FOLLOWER
I20260620 20:09:32.945678 7022 consensus_queue.cc:260] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:32.946904 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.936204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:38989 (local address 127.4.169.131:46277)
0620 20:09:32.936471 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.936499 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.936651 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.937603 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.937608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.937634 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:32.937893 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.937910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.940439 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.940448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.943080 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.943092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.945935 (+ 2843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.945945 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.946188 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.946283 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.946381 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.946575 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":49}
I20260620 20:09:32.946177 7028 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:32.947113 7028 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:32.947594 7022 ts_tablet_manager.cc:1428] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 20:09:32.947768 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.937010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:46277 (local address 127.4.169.130:38989)
0620 20:09:32.937197 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.937203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.937220 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:09:32.937277 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.937282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.937360 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.937446 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:32.938454 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.939979 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.944546 (+ 4567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.945776 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.945935 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.946007 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.947267 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.947431 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:09:32.948621 6818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5d185cbd9b4118ba3e233d7871e916" candidate_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d27450e651486bbf8db3ef6327a390" is_pre_election: true
I20260620 20:09:32.948860 6818 raft_consensus.cc:2454] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc8ebb583a1c4a598cd3ac04b6475e71 in term 0.
I20260620 20:09:32.949431 6884 leader_election.cc:302] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8d27450e651486bbf8db3ef6327a390, dc8ebb583a1c4a598cd3ac04b6475e71; no voters:
I20260620 20:09:32.949923 7045 raft_consensus.cc:2781] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:32.950088 7045 raft_consensus.cc:482] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:32.950207 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.937784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:37653 (local address 127.4.169.129:33775)
0620 20:09:32.938291 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.938297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.943169 (+ 4872us) server_negotiation.cc:407] Connection header received
0620 20:09:32.943344 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.943350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.943435 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.943521 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:32.944356 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.945510 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.948034 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.949085 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.949209 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.949255 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.949843 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.949967 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350}
I20260620 20:09:32.950286 7045 raft_consensus.cc:3037] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:32.950352 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.936940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:33775 (local address 127.4.169.131:37653)
0620 20:09:32.943035 (+ 6095us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.943060 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.943201 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.943676 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.943681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.943705 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:32.944136 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.944148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.945705 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.945711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.947835 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.947873 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.949467 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.949476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.949671 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.949712 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.949813 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.949961 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5937,"thread_start_us":5771,"threads_started":1}
I20260620 20:09:32.951303 6678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5d185cbd9b4118ba3e233d7871e916" candidate_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4174d7885645e896ca30bda3eec81d" is_pre_election: true
I20260620 20:09:32.951716 6678 raft_consensus.cc:2454] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc8ebb583a1c4a598cd3ac04b6475e71 in term 0.
I20260620 20:09:32.953066 7045 raft_consensus.cc:504] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:32.953733 7045 leader_election.cc:288] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [CANDIDATE]: Term 1 election: Requested vote from peers ee4174d7885645e896ca30bda3eec81d (127.4.169.129:33775), a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989)
I20260620 20:09:32.954049 6678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5d185cbd9b4118ba3e233d7871e916" candidate_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4174d7885645e896ca30bda3eec81d"
I20260620 20:09:32.954238 6678 raft_consensus.cc:3037] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:32.954245 6818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5d185cbd9b4118ba3e233d7871e916" candidate_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d27450e651486bbf8db3ef6327a390"
I20260620 20:09:32.954512 6818 raft_consensus.cc:3037] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:32.956322 6678 raft_consensus.cc:2454] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc8ebb583a1c4a598cd3ac04b6475e71 in term 1.
I20260620 20:09:32.956391 6818 raft_consensus.cc:2454] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc8ebb583a1c4a598cd3ac04b6475e71 in term 1.
I20260620 20:09:32.956710 6885 leader_election.cc:302] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc8ebb583a1c4a598cd3ac04b6475e71, ee4174d7885645e896ca30bda3eec81d; no voters:
I20260620 20:09:32.957049 7045 raft_consensus.cc:2781] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:32.957216 7045 raft_consensus.cc:686] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 LEADER]: Becoming Leader. State: Replica: dc8ebb583a1c4a598cd3ac04b6475e71, State: Running, Role: LEADER
I20260620 20:09:32.957664 7045 consensus_queue.cc:237] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:32.961205 7029 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: TxnStatusTablet state changed. Reason: New leader dc8ebb583a1c4a598cd3ac04b6475e71. Latest consensus state: current_term: 1 leader_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:32.961483 7029 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:09:32.962114 7053 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:09:32.962378 7053 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:09:32.962754 6515 catalog_manager.cc:5495] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131). New cstate: current_term: 1 leader_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } health_report { overall_health: HEALTHY } } }
I20260620 20:09:32.984509 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.978185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37031 (local address 127.0.0.1:47184)
0620 20:09:32.978553 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:32.978569 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:32.978678 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:32.979136 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:32.979140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:32.979149 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:09:32.979297 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.979304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.980811 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.980815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.981799 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:32.981810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.982858 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.982864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:32.983037 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.983093 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:09:32.984091 (+ 998us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:09:32.984107 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 20:09:32.984166 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 20:09:32.984276 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260620 20:09:32.984582 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:32.978344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.4.169.131:37031)
0620 20:09:32.978545 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:32.978552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:32.978635 (+ 83us) server_negotiation.cc:407] Connection header received
0620 20:09:32.978856 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:32.978864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:32.978964 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:32.979119 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:32.979442 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.980674 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.981952 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:32.982677 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:32.982827 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:32.983266 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:32.983983 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:32.984139 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:32.984200 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 20:09:32.984281 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:09:32.991715 6678 raft_consensus.cc:1261] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Refusing update from remote peer dc8ebb583a1c4a598cd3ac04b6475e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:32.991928 6818 raft_consensus.cc:1261] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Refusing update from remote peer dc8ebb583a1c4a598cd3ac04b6475e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:32.992650 7029 consensus_queue.cc:1035] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:32.993055 7045 consensus_queue.cc:1035] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:33.000638 7030 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: TxnStatusTablet state changed. Reason: New leader dc8ebb583a1c4a598cd3ac04b6475e71. Latest consensus state: current_term: 1 leader_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:33.000918 7030 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:33.005010 7028 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: TxnStatusTablet state changed. Reason: New leader dc8ebb583a1c4a598cd3ac04b6475e71. Latest consensus state: current_term: 1 leader_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:33.005318 7028 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:33.007566 7058 mvcc.cc:205] Tried to move back new op lower bound from 7299015364565282816 to 7299015364438388736. Current Snapshot: MvccSnapshot[applied={T|T < 7299015364565282816}]
I20260620 20:09:33.007566 7056 mvcc.cc:205] Tried to move back new op lower bound from 7299015364565282816 to 7299015364438388736. Current Snapshot: MvccSnapshot[applied={T|T < 7299015364565282816}]
I20260620 20:09:33.008767 7029 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:33.009127 7029 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:09:33.010941 7029 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:33.011209 7029 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P dc8ebb583a1c4a598cd3ac04b6475e71: This TxnStatusTablet replica's current role is: LEADER
W20260620 20:09:33.017923 6724 tablet.cc:2369] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:33.018976 7028 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:33.019236 7028 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P ee4174d7885645e896ca30bda3eec81d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:33.023341 7030 tablet_replica.cc:439] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } } }
I20260620 20:09:33.023679 7030 tablet_replica.cc:442] T aa5d185cbd9b4118ba3e233d7871e916 P a8d27450e651486bbf8db3ef6327a390: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:09:33.035894 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.027590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.4.169.129:33775)
0620 20:09:33.027754 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:33.027760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:33.027986 (+ 226us) server_negotiation.cc:407] Connection header received
0620 20:09:33.028214 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:33.028219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:33.028346 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:33.028492 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:33.028944 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.030224 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.032200 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.033207 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.033962 (+ 755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.034106 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.034879 (+ 773us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.035012 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:33.035469 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 20:09:33.035554 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:09:33.053086 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.046892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37031 (local address 127.4.169.129:54649)
0620 20:09:33.047542 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:33.047559 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:33.047671 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:33.047992 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:33.047995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:33.048004 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:09:33.048297 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.048304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.049708 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.049712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.050901 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.050914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.051929 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.051935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.052080 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.052144 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.052605 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.052612 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:09:33.052675 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 20:09:33.052784 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":317,"threads_started":1}
I20260620 20:09:33.053087 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.046982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:54649 (local address 127.4.169.131:37031)
0620 20:09:33.047194 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:33.047201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:33.047639 (+ 438us) server_negotiation.cc:407] Connection header received
0620 20:09:33.047786 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:33.047792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:33.047877 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:33.047976 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:33.048440 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.049571 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.051034 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.051723 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.051855 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.052370 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.052506 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.052603 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:33.052718 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:09:33.052801 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:09:33.062472 6818 raft_consensus.cc:1261] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Refusing update from remote peer ee4174d7885645e896ca30bda3eec81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:33.062949 6958 raft_consensus.cc:1261] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Refusing update from remote peer ee4174d7885645e896ca30bda3eec81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:33.063105 7033 consensus_queue.cc:1035] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:33.063733 7028 consensus_queue.cc:1035] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:33.099244 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.092622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:33775 (local address 127.4.169.131:43585)
0620 20:09:33.092940 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:33.092955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:33.093108 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:33.093457 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:33.093462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:33.093474 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:09:33.093666 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.093677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.095167 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.095173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.096915 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.096930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.097905 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.097912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.098086 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.098159 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.098658 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.098672 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 20:09:33.098750 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 20:09:33.098896 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260620 20:09:33.099881 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.092608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:43585 (local address 127.4.169.129:33775)
0620 20:09:33.092789 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:33.092793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:33.093037 (+ 244us) server_negotiation.cc:407] Connection header received
0620 20:09:33.093244 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:33.093249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:33.093327 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:33.093421 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:33.093854 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.095003 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.097068 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.097739 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.097878 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.098365 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.098527 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.098608 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:33.099524 (+ 916us) server_negotiation.cc:299] Negotiation successful
0620 20:09:33.099602 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:09:33.140107 6515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:33.140401 6515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:33.144904 6515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1: Sending DeleteTablet for 3 replicas of tablet b135985df1524cf98a8fc8a9a1bdcd28
I20260620 20:09:33.146150 6653 tablet_service.cc:1514] Processing DeleteTablet for tablet b135985df1524cf98a8fc8a9a1bdcd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:33 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:59188
I20260620 20:09:33.146885 6793 tablet_service.cc:1514] Processing DeleteTablet for tablet b135985df1524cf98a8fc8a9a1bdcd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:33 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:36296
I20260620 20:09:33.148892 6515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:09:33.150054 6515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:33.150712 6933 tablet_service.cc:1514] Processing DeleteTablet for tablet b135985df1524cf98a8fc8a9a1bdcd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:33 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:47176
I20260620 20:09:33.156981 7075 tablet_replica.cc:330] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: stopping tablet replica
I20260620 20:09:33.157564 7075 raft_consensus.cc:2229] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:09:33.158290 7075 raft_consensus.cc:2258] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:33.160269 7075 ts_tablet_manager.cc:1905] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:33.161073 6932 tablet_service.cc:1467] Processing CreateTablet for tablet 78021b91efa84cc6863e3965a0c03de8 (DEFAULT_TABLE table=test-table [id=6e476c91fdba4f62bc39516a210e435a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:33.161559 6932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78021b91efa84cc6863e3965a0c03de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:33.161484 6653 tablet_service.cc:1467] Processing CreateTablet for tablet 78021b91efa84cc6863e3965a0c03de8 (DEFAULT_TABLE table=test-table [id=6e476c91fdba4f62bc39516a210e435a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:33.162614 6792 tablet_service.cc:1467] Processing CreateTablet for tablet 78021b91efa84cc6863e3965a0c03de8 (DEFAULT_TABLE table=test-table [id=6e476c91fdba4f62bc39516a210e435a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:33.163152 6792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78021b91efa84cc6863e3965a0c03de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:33.168697 7023 tablet_bootstrap.cc:492] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: Bootstrap starting.
I20260620 20:09:33.170497 7023 tablet_bootstrap.cc:654] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:33.175452 6653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78021b91efa84cc6863e3965a0c03de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:33.180171 7026 tablet_bootstrap.cc:492] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: Bootstrap starting.
I20260620 20:09:33.182053 7026 tablet_bootstrap.cc:654] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:33.183183 7022 tablet_bootstrap.cc:492] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: Bootstrap starting.
I20260620 20:09:33.185286 7022 tablet_bootstrap.cc:654] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:33.189505 7075 ts_tablet_manager.cc:1918] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:09:33.189750 7075 log.cc:1196] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/b135985df1524cf98a8fc8a9a1bdcd28
I20260620 20:09:33.190426 7075 ts_tablet_manager.cc:1939] T b135985df1524cf98a8fc8a9a1bdcd28 P ee4174d7885645e896ca30bda3eec81d: Deleting consensus metadata
I20260620 20:09:33.191584 7023 tablet_bootstrap.cc:492] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: No bootstrap required, opened a new log
I20260620 20:09:33.191756 7023 ts_tablet_manager.cc:1397] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.002s
I20260620 20:09:33.192562 7023 raft_consensus.cc:348] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:33.192832 7023 raft_consensus.cc:374] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:33.192905 7023 raft_consensus.cc:729] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8ebb583a1c4a598cd3ac04b6475e71, State: Initialized, Role: FOLLOWER
I20260620 20:09:33.193126 7023 consensus_queue.cc:260] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:33.193663 7023 ts_tablet_manager.cc:1428] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 20:09:33.195993 6502 catalog_manager.cc:4841] TS ee4174d7885645e896ca30bda3eec81d (127.4.169.129:33775): tablet b135985df1524cf98a8fc8a9a1bdcd28 (table test-table [id=8dcfb6631d1e4c55ae11efb6412f5ac0]) successfully deleted
I20260620 20:09:33.210965 7026 tablet_bootstrap.cc:492] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: No bootstrap required, opened a new log
I20260620 20:09:33.214424 7026 ts_tablet_manager.cc:1397] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.006s
I20260620 20:09:33.215508 7026 raft_consensus.cc:348] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:33.216289 7077 tablet_replica.cc:330] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: stopping tablet replica
I20260620 20:09:33.216928 7022 tablet_bootstrap.cc:492] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: No bootstrap required, opened a new log
I20260620 20:09:33.217133 7022 ts_tablet_manager.cc:1397] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.009s
I20260620 20:09:33.217720 7022 raft_consensus.cc:348] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:33.217979 7022 raft_consensus.cc:374] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:33.218073 7022 raft_consensus.cc:729] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4174d7885645e896ca30bda3eec81d, State: Initialized, Role: FOLLOWER
I20260620 20:09:33.218338 7022 consensus_queue.cc:260] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:33.218962 7077 raft_consensus.cc:2229] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:33.219276 7077 raft_consensus.cc:2258] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:33.219604 7026 raft_consensus.cc:374] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:33.216845 7076 tablet_replica.cc:330] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: stopping tablet replica
I20260620 20:09:33.220520 7076 raft_consensus.cc:2229] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:33.220908 7076 raft_consensus.cc:2258] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:33.221040 7026 raft_consensus.cc:729] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d27450e651486bbf8db3ef6327a390, State: Initialized, Role: FOLLOWER
I20260620 20:09:33.221356 7077 ts_tablet_manager.cc:1905] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:33.221613 7026 consensus_queue.cc:260] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:33.225523 7026 ts_tablet_manager.cc:1428] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260620 20:09:33.228660 7022 ts_tablet_manager.cc:1428] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20260620 20:09:33.230176 7077 ts_tablet_manager.cc:1918] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:09:33.230373 7077 log.cc:1196] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/b135985df1524cf98a8fc8a9a1bdcd28
I20260620 20:09:33.231066 7077 ts_tablet_manager.cc:1939] T b135985df1524cf98a8fc8a9a1bdcd28 P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting consensus metadata
I20260620 20:09:33.232983 6500 catalog_manager.cc:4841] TS dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031): tablet b135985df1524cf98a8fc8a9a1bdcd28 (table test-table [id=8dcfb6631d1e4c55ae11efb6412f5ac0]) successfully deleted
I20260620 20:09:33.229610 7076 ts_tablet_manager.cc:1905] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:33.239928 7076 ts_tablet_manager.cc:1918] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:09:33.240113 7076 log.cc:1196] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/b135985df1524cf98a8fc8a9a1bdcd28
I20260620 20:09:33.240553 7076 ts_tablet_manager.cc:1939] T b135985df1524cf98a8fc8a9a1bdcd28 P a8d27450e651486bbf8db3ef6327a390: Deleting consensus metadata
I20260620 20:09:33.242053 6501 catalog_manager.cc:4841] TS a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989): tablet b135985df1524cf98a8fc8a9a1bdcd28 (table test-table [id=8dcfb6631d1e4c55ae11efb6412f5ac0]) successfully deleted
I20260620 20:09:33.477641 7030 raft_consensus.cc:482] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:33.477893 7030 raft_consensus.cc:504] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:33.479908 7030 leader_election.cc:288] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee4174d7885645e896ca30bda3eec81d (127.4.169.129:33775), dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031)
I20260620 20:09:33.487864 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.480562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:44379 (local address 127.4.169.131:37031)
0620 20:09:33.480775 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:33.480784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:33.481436 (+ 652us) server_negotiation.cc:407] Connection header received
0620 20:09:33.481672 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:33.481680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:33.481756 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:33.481853 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:33.482597 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.483602 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.485716 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.486919 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.487085 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.487182 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:33.487479 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 20:09:33.487616 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:09:33.487943 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.480193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37031 (local address 127.4.169.130:44379)
0620 20:09:33.481345 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:33.481367 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:33.481533 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:33.481939 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:33.481944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:33.481976 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:33.482411 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.482425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.483721 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.483729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.485552 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.485564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.487103 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.487110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.487293 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.487331 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:09:33.487437 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:09:33.487574 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":126,"thread_start_us":880,"threads_started":1}
I20260620 20:09:33.489015 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78021b91efa84cc6863e3965a0c03de8" candidate_uuid: "a8d27450e651486bbf8db3ef6327a390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" is_pre_election: true
I20260620 20:09:33.489288 6957 raft_consensus.cc:2454] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d27450e651486bbf8db3ef6327a390 in term 0.
I20260620 20:09:33.489387 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.480409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:36795 (local address 127.4.169.129:33775)
0620 20:09:33.480679 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:33.480688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:33.481177 (+ 489us) server_negotiation.cc:407] Connection header received
0620 20:09:33.481472 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:33.481480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:33.481574 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:33.481702 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:33.482185 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.483486 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.485969 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.487339 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.487532 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.487598 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:33.488959 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0620 20:09:33.489083 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 20:09:33.489444 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.480036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:33775 (local address 127.4.169.130:36795)
0620 20:09:33.481046 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:33.481070 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:33.481291 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:33.481707 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:33.481711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:33.481744 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:33.482019 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.482034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.483635 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.483641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.485800 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.485814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.488595 (+ 2781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.488602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.488783 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.488819 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:09:33.488935 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 20:09:33.489067 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":3,"thread_start_us":784,"threads_started":1}
I20260620 20:09:33.489990 6678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78021b91efa84cc6863e3965a0c03de8" candidate_uuid: "a8d27450e651486bbf8db3ef6327a390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4174d7885645e896ca30bda3eec81d" is_pre_election: true
I20260620 20:09:33.490151 6743 leader_election.cc:302] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8d27450e651486bbf8db3ef6327a390, dc8ebb583a1c4a598cd3ac04b6475e71; no voters:
I20260620 20:09:33.490243 6678 raft_consensus.cc:2454] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d27450e651486bbf8db3ef6327a390 in term 0.
I20260620 20:09:33.490679 7030 raft_consensus.cc:2781] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:33.490811 7030 raft_consensus.cc:482] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:33.490917 7030 raft_consensus.cc:3037] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:33.492988 7030 raft_consensus.cc:504] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:33.493630 7030 leader_election.cc:288] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [CANDIDATE]: Term 1 election: Requested vote from peers ee4174d7885645e896ca30bda3eec81d (127.4.169.129:33775), dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031)
I20260620 20:09:33.494037 6678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78021b91efa84cc6863e3965a0c03de8" candidate_uuid: "a8d27450e651486bbf8db3ef6327a390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4174d7885645e896ca30bda3eec81d"
I20260620 20:09:33.494197 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78021b91efa84cc6863e3965a0c03de8" candidate_uuid: "a8d27450e651486bbf8db3ef6327a390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71"
I20260620 20:09:33.494271 6678 raft_consensus.cc:3037] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:33.494428 6957 raft_consensus.cc:3037] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:33.497335 6957 raft_consensus.cc:2454] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8d27450e651486bbf8db3ef6327a390 in term 1.
I20260620 20:09:33.497565 6678 raft_consensus.cc:2454] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8d27450e651486bbf8db3ef6327a390 in term 1.
I20260620 20:09:33.497779 6743 leader_election.cc:302] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8d27450e651486bbf8db3ef6327a390, dc8ebb583a1c4a598cd3ac04b6475e71; no voters:
I20260620 20:09:33.498059 7030 raft_consensus.cc:2781] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:33.498600 7030 raft_consensus.cc:686] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 1 LEADER]: Becoming Leader. State: Replica: a8d27450e651486bbf8db3ef6327a390, State: Running, Role: LEADER
I20260620 20:09:33.499240 7030 consensus_queue.cc:237] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:33.502962 6515 catalog_manager.cc:5495] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8d27450e651486bbf8db3ef6327a390 (127.4.169.130). New cstate: current_term: 1 leader_uuid: "a8d27450e651486bbf8db3ef6327a390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } health_report { overall_health: UNKNOWN } } }
I20260620 20:09:33.574486 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.563646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.4.169.130:38989)
0620 20:09:33.568064 (+ 4418us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:33.568069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:33.568087 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:09:33.568160 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:33.568164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:33.568248 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:33.568384 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:33.568947 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.569745 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.571750 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.572160 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.572289 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.572742 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.573356 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.573462 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:33.573920 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 20:09:33.573989 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4310,"thread_start_us":4184,"threads_started":1}
I20260620 20:09:33.590776 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.584911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:50761 (local address 127.4.169.131:37031)
0620 20:09:33.585102 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:33.585108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:33.585398 (+ 290us) server_negotiation.cc:407] Connection header received
0620 20:09:33.585562 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:33.585566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:33.585628 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:33.585700 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:33.586278 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.587095 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.589171 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.589556 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.589642 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.590129 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.590229 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.590291 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:33.590521 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 20:09:33.590596 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:09:33.591018 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.584718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37031 (local address 127.4.169.130:50761)
0620 20:09:33.585299 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:33.585319 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:33.585453 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:33.585734 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:33.585741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:33.585756 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:09:33.586097 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.586107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.587364 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.587370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.589047 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.589061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.589721 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.589727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.589907 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.589967 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.590326 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.590347 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:09:33.590465 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:09:33.590632 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":239,"threads_started":1}
I20260620 20:09:33.599594 6957 raft_consensus.cc:1261] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Refusing update from remote peer a8d27450e651486bbf8db3ef6327a390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:33.599702 6678 raft_consensus.cc:1261] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Refusing update from remote peer a8d27450e651486bbf8db3ef6327a390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:33.600306 7083 consensus_queue.cc:1035] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:33.601018 7083 consensus_queue.cc:1035] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:33.633486 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.628640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:33767 (local address 127.4.169.130:38989)
0620 20:09:33.628821 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:33.628827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:33.628982 (+ 155us) server_negotiation.cc:407] Connection header received
0620 20:09:33.629175 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:33.629180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:33.629263 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:33.629359 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:33.629760 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.630691 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.631899 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.632313 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.632450 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.632815 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.632922 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.633019 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:33.633127 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 20:09:33.633203 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:09:33.633551 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:33.628370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:38989 (local address 127.4.169.131:33767)
0620 20:09:33.628880 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:33.628898 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:33.629044 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:33.629372 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:33.629376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:33.629387 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:09:33.629589 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.629597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.630839 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.630843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.631753 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:33.631763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:33.632468 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:33.632474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:33.632623 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:33.632659 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:09:33.633019 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:09:33.633033 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 20:09:33.633110 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 20:09:33.633257 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":1277,"threads_started":1}
W20260620 20:09:33.634996 7057 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:09:35.229898 6516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:35.230103 6516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:35.234012 6516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1: Sending DeleteTablet for 3 replicas of tablet 78021b91efa84cc6863e3965a0c03de8
I20260620 20:09:35.234927 6793 tablet_service.cc:1514] Processing DeleteTablet for tablet 78021b91efa84cc6863e3965a0c03de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:35 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:36296
I20260620 20:09:35.235459 6653 tablet_service.cc:1514] Processing DeleteTablet for tablet 78021b91efa84cc6863e3965a0c03de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:35 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:59188
I20260620 20:09:35.235705 6932 tablet_service.cc:1514] Processing DeleteTablet for tablet 78021b91efa84cc6863e3965a0c03de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:35 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:47176
I20260620 20:09:35.235914 7120 tablet_replica.cc:330] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: stopping tablet replica
I20260620 20:09:35.236100 7121 tablet_replica.cc:330] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: stopping tablet replica
I20260620 20:09:35.236137 7120 raft_consensus.cc:2229] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:09:35.236452 7121 raft_consensus.cc:2229] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:35.236723 7120 raft_consensus.cc:2258] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:35.237964 7121 raft_consensus.cc:2258] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:35.239375 7121 ts_tablet_manager.cc:1905] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:35.242467 7122 tablet_replica.cc:330] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: stopping tablet replica
I20260620 20:09:35.242765 7122 raft_consensus.cc:2229] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:35.242893 7120 ts_tablet_manager.cc:1905] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:35.243108 7122 raft_consensus.cc:2258] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:35.244272 7122 ts_tablet_manager.cc:1905] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:35.246771 7121 ts_tablet_manager.cc:1918] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:09:35.246912 7121 log.cc:1196] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/78021b91efa84cc6863e3965a0c03de8
I20260620 20:09:35.247454 7121 ts_tablet_manager.cc:1939] T 78021b91efa84cc6863e3965a0c03de8 P ee4174d7885645e896ca30bda3eec81d: Deleting consensus metadata
I20260620 20:09:35.248632 6502 catalog_manager.cc:4841] TS ee4174d7885645e896ca30bda3eec81d (127.4.169.129:33775): tablet 78021b91efa84cc6863e3965a0c03de8 (table test-table [id=6e476c91fdba4f62bc39516a210e435a]) successfully deleted
I20260620 20:09:35.249522 7120 ts_tablet_manager.cc:1918] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:09:35.249665 7120 log.cc:1196] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/78021b91efa84cc6863e3965a0c03de8
I20260620 20:09:35.250067 7120 ts_tablet_manager.cc:1939] T 78021b91efa84cc6863e3965a0c03de8 P a8d27450e651486bbf8db3ef6327a390: Deleting consensus metadata
I20260620 20:09:35.251024 6501 catalog_manager.cc:4841] TS a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989): tablet 78021b91efa84cc6863e3965a0c03de8 (table test-table [id=6e476c91fdba4f62bc39516a210e435a]) successfully deleted
I20260620 20:09:35.251724 7122 ts_tablet_manager.cc:1918] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:09:35.251864 7122 log.cc:1196] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/78021b91efa84cc6863e3965a0c03de8
I20260620 20:09:35.252306 7122 ts_tablet_manager.cc:1939] T 78021b91efa84cc6863e3965a0c03de8 P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting consensus metadata
I20260620 20:09:35.253373 6500 catalog_manager.cc:4841] TS dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031): tablet 78021b91efa84cc6863e3965a0c03de8 (table test-table [id=6e476c91fdba4f62bc39516a210e435a]) successfully deleted
Jun 20 20:09:35 dist-test-slave-4bwq krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:09:35 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:35.290283 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:35.276753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.4.169.190:40449)
0620 20:09:35.277348 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:35.277354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:35.277470 (+ 116us) server_negotiation.cc:407] Connection header received
0620 20:09:35.277685 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:35.277690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:35.277751 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:35.277865 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:35.279168 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:35.279929 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:35.281161 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:35.281563 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:35.281707 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:35.285431 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:35.285436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:35.285440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:35.285472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:35.288089 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:35.288647 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:35.288651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:35.288653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:35.288781 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:35.289073 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:35.289076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:35.289078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:35.289372 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:35.289539 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:35.289748 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 20:09:35.289904 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":272,"threads_started":1}
I20260620 20:09:35.293378 6516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:09:35.293893 6516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:35.299822 6653 tablet_service.cc:1467] Processing CreateTablet for tablet 5075a34b80fd4543ae3f426dab88b797 (DEFAULT_TABLE table=test-table [id=422b618058d94f85a1b21e7b5a55e54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:35.299867 6793 tablet_service.cc:1467] Processing CreateTablet for tablet 5075a34b80fd4543ae3f426dab88b797 (DEFAULT_TABLE table=test-table [id=422b618058d94f85a1b21e7b5a55e54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:35.300032 6932 tablet_service.cc:1467] Processing CreateTablet for tablet 5075a34b80fd4543ae3f426dab88b797 (DEFAULT_TABLE table=test-table [id=422b618058d94f85a1b21e7b5a55e54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:35.300253 6653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075a34b80fd4543ae3f426dab88b797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:35.300280 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075a34b80fd4543ae3f426dab88b797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:35.300438 6932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075a34b80fd4543ae3f426dab88b797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:35.306412 7131 tablet_bootstrap.cc:492] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: Bootstrap starting.
I20260620 20:09:35.307152 7133 tablet_bootstrap.cc:492] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: Bootstrap starting.
I20260620 20:09:35.308012 7132 tablet_bootstrap.cc:492] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: Bootstrap starting.
I20260620 20:09:35.308230 7131 tablet_bootstrap.cc:654] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:35.308902 7133 tablet_bootstrap.cc:654] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:35.310250 7132 tablet_bootstrap.cc:654] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:35.310994 7131 tablet_bootstrap.cc:492] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: No bootstrap required, opened a new log
I20260620 20:09:35.311157 7131 ts_tablet_manager.cc:1397] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 20:09:35.311487 7133 tablet_bootstrap.cc:492] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: No bootstrap required, opened a new log
I20260620 20:09:35.311589 7133 ts_tablet_manager.cc:1397] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:09:35.311980 7131 raft_consensus.cc:348] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.312192 7131 raft_consensus.cc:374] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:35.312284 7131 raft_consensus.cc:729] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d27450e651486bbf8db3ef6327a390, State: Initialized, Role: FOLLOWER
I20260620 20:09:35.312350 7132 tablet_bootstrap.cc:492] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: No bootstrap required, opened a new log
I20260620 20:09:35.312537 7132 ts_tablet_manager.cc:1397] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:09:35.312383 7133 raft_consensus.cc:348] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.312464 7131 consensus_queue.cc:260] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.312629 7133 raft_consensus.cc:374] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:35.312716 7133 raft_consensus.cc:729] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8ebb583a1c4a598cd3ac04b6475e71, State: Initialized, Role: FOLLOWER
I20260620 20:09:35.312994 7131 ts_tablet_manager.cc:1428] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 20:09:35.312932 7133 consensus_queue.cc:260] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.313344 7132 raft_consensus.cc:348] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.313563 7133 ts_tablet_manager.cc:1428] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 20:09:35.313584 7132 raft_consensus.cc:374] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:35.313678 7132 raft_consensus.cc:729] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4174d7885645e896ca30bda3eec81d, State: Initialized, Role: FOLLOWER
I20260620 20:09:35.313936 7132 consensus_queue.cc:260] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.315080 7132 ts_tablet_manager.cc:1428] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:09:35.449673 7137 raft_consensus.cc:482] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:35.449892 7137 raft_consensus.cc:504] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.450591 7137 leader_election.cc:288] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989), dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031)
I20260620 20:09:35.451176 6818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075a34b80fd4543ae3f426dab88b797" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d27450e651486bbf8db3ef6327a390" is_pre_election: true
I20260620 20:09:35.451179 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075a34b80fd4543ae3f426dab88b797" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" is_pre_election: true
I20260620 20:09:35.451409 6818 raft_consensus.cc:2454] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 0.
I20260620 20:09:35.451423 6957 raft_consensus.cc:2454] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 0.
I20260620 20:09:35.451896 6603 leader_election.cc:302] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc8ebb583a1c4a598cd3ac04b6475e71, ee4174d7885645e896ca30bda3eec81d; no voters:
I20260620 20:09:35.452189 7137 raft_consensus.cc:2781] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:35.452309 7137 raft_consensus.cc:482] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:35.452404 7137 raft_consensus.cc:3037] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:35.454856 7137 raft_consensus.cc:504] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.455503 7137 leader_election.cc:288] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 election: Requested vote from peers a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989), dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031)
I20260620 20:09:35.455839 6818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075a34b80fd4543ae3f426dab88b797" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d27450e651486bbf8db3ef6327a390"
I20260620 20:09:35.455907 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075a34b80fd4543ae3f426dab88b797" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71"
I20260620 20:09:35.456055 6818 raft_consensus.cc:3037] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:35.456092 6957 raft_consensus.cc:3037] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:35.458081 6957 raft_consensus.cc:2454] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 1.
I20260620 20:09:35.458106 6818 raft_consensus.cc:2454] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 1.
I20260620 20:09:35.458515 6603 leader_election.cc:302] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc8ebb583a1c4a598cd3ac04b6475e71, ee4174d7885645e896ca30bda3eec81d; no voters:
I20260620 20:09:35.458814 7137 raft_consensus.cc:2781] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:35.459321 7137 raft_consensus.cc:686] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 1 LEADER]: Becoming Leader. State: Replica: ee4174d7885645e896ca30bda3eec81d, State: Running, Role: LEADER
I20260620 20:09:35.459678 7137 consensus_queue.cc:237] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.462224 6515 catalog_manager.cc:5495] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee4174d7885645e896ca30bda3eec81d (127.4.169.129). New cstate: current_term: 1 leader_uuid: "ee4174d7885645e896ca30bda3eec81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } health_report { overall_health: UNKNOWN } } }
I20260620 20:09:35.508973 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:35.503106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.4.169.129:33775)
0620 20:09:35.503741 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:35.503745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:35.503764 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:09:35.503812 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:35.503816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:35.503873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:35.503972 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:35.504400 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:35.505157 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:35.506501 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:35.507452 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:35.507586 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:35.508086 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:35.508215 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:35.508325 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:35.508505 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 20:09:35.508581 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":387,"threads_started":1}
I20260620 20:09:35.518464 6957 raft_consensus.cc:1261] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Refusing update from remote peer ee4174d7885645e896ca30bda3eec81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:35.518500 6818 raft_consensus.cc:1261] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Refusing update from remote peer ee4174d7885645e896ca30bda3eec81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:35.519042 7137 consensus_queue.cc:1035] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:35.519590 7138 consensus_queue.cc:1035] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:35.569142 6516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:35.569350 6516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:35.572284 6516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1: Sending DeleteTablet for 3 replicas of tablet 5075a34b80fd4543ae3f426dab88b797
I20260620 20:09:35.573064 6653 tablet_service.cc:1514] Processing DeleteTablet for tablet 5075a34b80fd4543ae3f426dab88b797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:35 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:59188
I20260620 20:09:35.573186 6793 tablet_service.cc:1514] Processing DeleteTablet for tablet 5075a34b80fd4543ae3f426dab88b797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:35 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:36296
I20260620 20:09:35.573319 6932 tablet_service.cc:1514] Processing DeleteTablet for tablet 5075a34b80fd4543ae3f426dab88b797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:35 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:47176
I20260620 20:09:35.573472 7121 tablet_replica.cc:330] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: stopping tablet replica
I20260620 20:09:35.573542 7120 tablet_replica.cc:330] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: stopping tablet replica
I20260620 20:09:35.573575 7122 tablet_replica.cc:330] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: stopping tablet replica
I20260620 20:09:35.573679 7121 raft_consensus.cc:2229] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:09:35.573760 7120 raft_consensus.cc:2229] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:35.573760 7122 raft_consensus.cc:2229] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:35.573935 7122 raft_consensus.cc:2258] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:35.573936 7120 raft_consensus.cc:2258] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:35.574002 7121 raft_consensus.cc:2258] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:35.574156 6516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:09:35.574898 6516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:35.575155 7120 ts_tablet_manager.cc:1905] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:35.575143 7122 ts_tablet_manager.cc:1905] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:35.575420 7121 ts_tablet_manager.cc:1905] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:35.581585 7120 ts_tablet_manager.cc:1918] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:09:35.581670 7121 ts_tablet_manager.cc:1918] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:09:35.581754 7120 log.cc:1196] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/5075a34b80fd4543ae3f426dab88b797
I20260620 20:09:35.581826 7121 log.cc:1196] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/5075a34b80fd4543ae3f426dab88b797
I20260620 20:09:35.581869 6653 tablet_service.cc:1467] Processing CreateTablet for tablet b0ce68ddd2ff44c5adf6b7097b2eb23f (DEFAULT_TABLE table=test-table [id=fb176b77cdda45a5ae5d54f9c4869aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:35.582197 7120 ts_tablet_manager.cc:1939] T 5075a34b80fd4543ae3f426dab88b797 P a8d27450e651486bbf8db3ef6327a390: Deleting consensus metadata
I20260620 20:09:35.582302 6653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ce68ddd2ff44c5adf6b7097b2eb23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:35.582233 6793 tablet_service.cc:1467] Processing CreateTablet for tablet b0ce68ddd2ff44c5adf6b7097b2eb23f (DEFAULT_TABLE table=test-table [id=fb176b77cdda45a5ae5d54f9c4869aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:35.582695 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ce68ddd2ff44c5adf6b7097b2eb23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:35.582882 7122 ts_tablet_manager.cc:1918] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:09:35.583491 6932 tablet_service.cc:1467] Processing CreateTablet for tablet b0ce68ddd2ff44c5adf6b7097b2eb23f (DEFAULT_TABLE table=test-table [id=fb176b77cdda45a5ae5d54f9c4869aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:35.583837 6932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ce68ddd2ff44c5adf6b7097b2eb23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:35.584189 7122 log.cc:1196] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/5075a34b80fd4543ae3f426dab88b797
I20260620 20:09:35.584668 7122 ts_tablet_manager.cc:1939] T 5075a34b80fd4543ae3f426dab88b797 P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting consensus metadata
I20260620 20:09:35.585615 6500 catalog_manager.cc:4841] TS dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031): tablet 5075a34b80fd4543ae3f426dab88b797 (table test-table [id=422b618058d94f85a1b21e7b5a55e54e]) successfully deleted
I20260620 20:09:35.586213 6501 catalog_manager.cc:4841] TS a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989): tablet 5075a34b80fd4543ae3f426dab88b797 (table test-table [id=422b618058d94f85a1b21e7b5a55e54e]) successfully deleted
I20260620 20:09:35.582293 7121 ts_tablet_manager.cc:1939] T 5075a34b80fd4543ae3f426dab88b797 P ee4174d7885645e896ca30bda3eec81d: Deleting consensus metadata
I20260620 20:09:35.587579 6502 catalog_manager.cc:4841] TS ee4174d7885645e896ca30bda3eec81d (127.4.169.129:33775): tablet 5075a34b80fd4543ae3f426dab88b797 (table test-table [id=422b618058d94f85a1b21e7b5a55e54e]) successfully deleted
I20260620 20:09:35.588852 7132 tablet_bootstrap.cc:492] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: Bootstrap starting.
I20260620 20:09:35.590070 7133 tablet_bootstrap.cc:492] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: Bootstrap starting.
I20260620 20:09:35.590473 7132 tablet_bootstrap.cc:654] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:35.590557 7131 tablet_bootstrap.cc:492] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: Bootstrap starting.
I20260620 20:09:35.591615 7133 tablet_bootstrap.cc:654] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:35.592859 7131 tablet_bootstrap.cc:654] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:35.593771 7133 tablet_bootstrap.cc:492] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: No bootstrap required, opened a new log
I20260620 20:09:35.593917 7133 ts_tablet_manager.cc:1397] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 20:09:35.594425 7132 tablet_bootstrap.cc:492] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: No bootstrap required, opened a new log
I20260620 20:09:35.594579 7132 ts_tablet_manager.cc:1397] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 20:09:35.594698 7133 raft_consensus.cc:348] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.594928 7133 raft_consensus.cc:374] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:35.595005 7133 raft_consensus.cc:729] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8ebb583a1c4a598cd3ac04b6475e71, State: Initialized, Role: FOLLOWER
I20260620 20:09:35.595253 7133 consensus_queue.cc:260] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.595328 7132 raft_consensus.cc:348] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.595790 7133 ts_tablet_manager.cc:1428] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:09:35.596197 7131 tablet_bootstrap.cc:492] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: No bootstrap required, opened a new log
I20260620 20:09:35.596352 7131 ts_tablet_manager.cc:1397] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 20:09:35.597002 7131 raft_consensus.cc:348] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.597251 7131 raft_consensus.cc:374] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:35.597340 7131 raft_consensus.cc:729] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d27450e651486bbf8db3ef6327a390, State: Initialized, Role: FOLLOWER
I20260620 20:09:35.597570 7131 consensus_queue.cc:260] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.598099 7131 ts_tablet_manager.cc:1428] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 20:09:35.598338 7132 raft_consensus.cc:374] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:35.598464 7132 raft_consensus.cc:729] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4174d7885645e896ca30bda3eec81d, State: Initialized, Role: FOLLOWER
I20260620 20:09:35.598687 7132 consensus_queue.cc:260] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.599162 7132 ts_tablet_manager.cc:1428] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 20:09:35.602952 7138 raft_consensus.cc:482] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:35.603138 7138 raft_consensus.cc:504] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.603645 7138 leader_election.cc:288] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989), dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031)
I20260620 20:09:35.604103 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ce68ddd2ff44c5adf6b7097b2eb23f" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d27450e651486bbf8db3ef6327a390" is_pre_election: true
I20260620 20:09:35.604261 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ce68ddd2ff44c5adf6b7097b2eb23f" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" is_pre_election: true
I20260620 20:09:35.604408 6817 raft_consensus.cc:2454] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 0.
I20260620 20:09:35.604496 6957 raft_consensus.cc:2454] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 0.
I20260620 20:09:35.604779 6604 leader_election.cc:302] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8d27450e651486bbf8db3ef6327a390, ee4174d7885645e896ca30bda3eec81d; no voters:
I20260620 20:09:35.605127 7138 raft_consensus.cc:2781] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:35.605247 7138 raft_consensus.cc:482] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:35.605369 7138 raft_consensus.cc:3037] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:35.607826 7138 raft_consensus.cc:504] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.608351 7138 leader_election.cc:288] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 election: Requested vote from peers a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989), dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031)
I20260620 20:09:35.608775 6817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ce68ddd2ff44c5adf6b7097b2eb23f" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d27450e651486bbf8db3ef6327a390"
I20260620 20:09:35.608991 6817 raft_consensus.cc:3037] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:35.608951 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ce68ddd2ff44c5adf6b7097b2eb23f" candidate_uuid: "ee4174d7885645e896ca30bda3eec81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71"
I20260620 20:09:35.609133 6957 raft_consensus.cc:3037] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:35.611194 6817 raft_consensus.cc:2454] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 1.
I20260620 20:09:35.611291 6957 raft_consensus.cc:2454] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4174d7885645e896ca30bda3eec81d in term 1.
I20260620 20:09:35.611578 6604 leader_election.cc:302] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8d27450e651486bbf8db3ef6327a390, ee4174d7885645e896ca30bda3eec81d; no voters:
I20260620 20:09:35.611832 7138 raft_consensus.cc:2781] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:35.611984 7138 raft_consensus.cc:686] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 1 LEADER]: Becoming Leader. State: Replica: ee4174d7885645e896ca30bda3eec81d, State: Running, Role: LEADER
I20260620 20:09:35.612221 7138 consensus_queue.cc:237] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } }
I20260620 20:09:35.614457 6515 catalog_manager.cc:5495] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee4174d7885645e896ca30bda3eec81d (127.4.169.129). New cstate: current_term: 1 leader_uuid: "ee4174d7885645e896ca30bda3eec81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4174d7885645e896ca30bda3eec81d" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 33775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 } health_report { overall_health: UNKNOWN } } }
I20260620 20:09:35.646535 6957 raft_consensus.cc:1261] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Refusing update from remote peer ee4174d7885645e896ca30bda3eec81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:35.646534 6817 raft_consensus.cc:1261] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Refusing update from remote peer ee4174d7885645e896ca30bda3eec81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:35.647086 7137 consensus_queue.cc:1035] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc8ebb583a1c4a598cd3ac04b6475e71" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:35.647418 7138 consensus_queue.cc:1035] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d27450e651486bbf8db3ef6327a390" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:09:35.668994 7104 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:09:37.249872 6515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:37.250070 6515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:37.253683 6515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 298a221c29594d04ae6252ceca4174a1: Sending DeleteTablet for 3 replicas of tablet b0ce68ddd2ff44c5adf6b7097b2eb23f
I20260620 20:09:37.254441 6653 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ce68ddd2ff44c5adf6b7097b2eb23f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:37 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:59188
I20260620 20:09:37.254554 6793 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ce68ddd2ff44c5adf6b7097b2eb23f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:37 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:36296
I20260620 20:09:37.254627 6932 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ce68ddd2ff44c5adf6b7097b2eb23f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:37 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:47176
I20260620 20:09:37.255611 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6587
I20260620 20:09:37.267545 7175 tablet_replica.cc:330] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: stopping tablet replica
I20260620 20:09:37.267833 7175 raft_consensus.cc:2229] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:09:37.268376 7175 raft_consensus.cc:2258] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:37.269063 7176 tablet_replica.cc:330] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: stopping tablet replica
I20260620 20:09:37.269282 7176 raft_consensus.cc:2229] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:37.269523 7176 raft_consensus.cc:2258] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:37.270458 7175 ts_tablet_manager.cc:1905] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:37.271030 7176 ts_tablet_manager.cc:1905] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:37.277449 7175 ts_tablet_manager.cc:1918] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:09:37.277812 7176 ts_tablet_manager.cc:1918] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:09:37.277961 7175 log.cc:1196] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/b0ce68ddd2ff44c5adf6b7097b2eb23f
I20260620 20:09:37.278034 7176 log.cc:1196] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/b0ce68ddd2ff44c5adf6b7097b2eb23f
I20260620 20:09:37.278559 7176 ts_tablet_manager.cc:1939] T b0ce68ddd2ff44c5adf6b7097b2eb23f P a8d27450e651486bbf8db3ef6327a390: Deleting consensus metadata
I20260620 20:09:37.278560 7175 ts_tablet_manager.cc:1939] T b0ce68ddd2ff44c5adf6b7097b2eb23f P ee4174d7885645e896ca30bda3eec81d: Deleting consensus metadata
I20260620 20:09:37.279771 6501 catalog_manager.cc:4841] TS a8d27450e651486bbf8db3ef6327a390 (127.4.169.130:38989): tablet b0ce68ddd2ff44c5adf6b7097b2eb23f (table test-table [id=fb176b77cdda45a5ae5d54f9c4869aea]) successfully deleted
I20260620 20:09:37.279973 6502 catalog_manager.cc:4841] TS ee4174d7885645e896ca30bda3eec81d (127.4.169.129:33775): tablet b0ce68ddd2ff44c5adf6b7097b2eb23f (table test-table [id=fb176b77cdda45a5ae5d54f9c4869aea]) successfully deleted
I20260620 20:09:37.281068 7177 tablet_replica.cc:330] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: stopping tablet replica
I20260620 20:09:37.281311 7177 raft_consensus.cc:2229] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:37.281550 7177 raft_consensus.cc:2258] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:37.282469 7177 ts_tablet_manager.cc:1905] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:37.287837 7177 ts_tablet_manager.cc:1918] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:09:37.287953 7177 log.cc:1196] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/b0ce68ddd2ff44c5adf6b7097b2eb23f
I20260620 20:09:37.288338 7177 ts_tablet_manager.cc:1939] T b0ce68ddd2ff44c5adf6b7097b2eb23f P dc8ebb583a1c4a598cd3ac04b6475e71: Deleting consensus metadata
I20260620 20:09:37.289363 6500 catalog_manager.cc:4841] TS dc8ebb583a1c4a598cd3ac04b6475e71 (127.4.169.131:37031): tablet b0ce68ddd2ff44c5adf6b7097b2eb23f (table test-table [id=fb176b77cdda45a5ae5d54f9c4869aea]) successfully deleted
Jun 20 20:09:37 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:37.299737 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:37.257700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.4.169.129:33775)
0620 20:09:37.279873 (+ 22173us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:37.279881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:37.280435 (+ 554us) server_negotiation.cc:407] Connection header received
0620 20:09:37.280652 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:37.280660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:37.280761 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:37.280945 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:37.283058 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:37.284139 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:37.285487 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:37.286138 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:37.286273 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:37.294572 (+ 8299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:37.294587 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:37.294603 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:37.294661 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:37.297346 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:37.297971 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:37.297978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:37.297980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:37.298107 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:37.298478 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:37.298481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:37.298484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:37.298837 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:37.299051 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:37.299287 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 20:09:37.299456 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22037,"thread_start_us":21924,"threads_started":1}
W20260620 20:09:37.300379 6718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59206
W20260620 20:09:37.301035 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6587: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:09:37.301184 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 6587
I20260620 20:09:37.323563 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6727
Jun 20 20:09:37 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:09:37.338742 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:37.324257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.4.169.130:38989)
0620 20:09:37.324765 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:37.324769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:37.324785 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:09:37.324837 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:37.324841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:37.324897 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:37.324987 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:37.326140 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:37.326926 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:37.328139 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:37.328535 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:37.328623 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:37.332702 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:37.332717 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:37.332738 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:37.332772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:37.335877 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:37.336612 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:37.336629 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:37.336632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:37.336747 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:37.337104 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:37.337110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:37.337113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:37.337543 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:37.337833 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:37.338121 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 20:09:37.338406 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":287,"threads_started":1}
W20260620 20:09:37.339392 6858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36304
W20260620 20:09:37.339977 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6727: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:09:37.340126 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 6727
I20260620 20:09:37.361677 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6867
Jun 20 20:09:37 dist-test-slave-4bwq krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:09:37.376621 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:37.362463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.4.169.131:37031)
0620 20:09:37.362988 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:37.362992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:37.363008 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:09:37.363060 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:37.363064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:37.363126 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:37.363256 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:37.364451 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:37.365207 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:37.366374 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:37.366852 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:37.366995 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:37.371230 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:37.371246 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:37.371262 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:37.371294 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:37.374192 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:37.374883 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:37.374889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:37.374892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:37.375003 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:37.375314 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:37.375317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:37.375319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:37.375629 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:37.375873 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:37.376048 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 20:09:37.376240 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
W20260620 20:09:37.377354 6998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47190
W20260620 20:09:37.377982 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6867: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:09:37.378129 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 6867
I20260620 20:09:37.398708 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 6484
I20260620 20:09:37.399866 6556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:37.515688 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 6484
2026-06-20T20:09:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10439 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:09:37 dist-test-slave-4bwq krb5kdc[7185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:09:37 dist-test-slave-4bwq krb5kdc[7185](info): set up 2 sockets
Jun 20 20:09:37 dist-test-slave-4bwq krb5kdc[7185](info): commencing operation
krb5kdc: starting...
W20260620 20:09:38.599295 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:09:38 dist-test-slave-4bwq krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:09:38.788026 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:37203
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:37203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:39.110280 7198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:39.110641 7198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:39.110740 7198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:39.117990 7198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:39.118098 7198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:39.118146 7198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:39.118209 7198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:39.118251 7198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:39.128463 7198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:37203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:37203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:39.129730 7198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:39.130968 7198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:39.142716 7206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:39.142797 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:39.143270 7203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:39.143597 7198 server_base.cc:1034] running on GCE node
I20260620 20:09:39.144372 7198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:39.145758 7198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:39.146991 7198 hybrid_clock.cc:648] HybridClock initialized: now 1781986179146947 us; error 60 us; skew 500 ppm
I20260620 20:09:39.147363 7198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:39 dist-test-slave-4bwq krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986179, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:39.150337 7198 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:09:39.151723 7198 webserver.cc:460] Webserver started at http://127.4.169.190:44137/ using document root <none> and password file <none>
I20260620 20:09:39.152393 7198 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:39.152532 7198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:39.152837 7198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:39.155819 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "5cb453601aeb43ce869db82fcf72f2cd"
format_stamp: "Formatted at 2026-06-20 20:09:39 on dist-test-slave-4bwq"
server_key: "84959271cf11257fb4162a2d4fe13576"
server_key_iv: "2050c4ce3bd850ba493830b8a7576993"
server_key_version: "encryptionkey@0"
I20260620 20:09:39.156582 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "5cb453601aeb43ce869db82fcf72f2cd"
format_stamp: "Formatted at 2026-06-20 20:09:39 on dist-test-slave-4bwq"
server_key: "84959271cf11257fb4162a2d4fe13576"
server_key_iv: "2050c4ce3bd850ba493830b8a7576993"
server_key_version: "encryptionkey@0"
I20260620 20:09:39.161535 7198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:09:39.164937 7213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:39.166281 7198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:09:39.166491 7198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "5cb453601aeb43ce869db82fcf72f2cd"
format_stamp: "Formatted at 2026-06-20 20:09:39 on dist-test-slave-4bwq"
server_key: "84959271cf11257fb4162a2d4fe13576"
server_key_iv: "2050c4ce3bd850ba493830b8a7576993"
server_key_version: "encryptionkey@0"
I20260620 20:09:39.166710 7198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:39.189843 7198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:39.193761 7198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:39.194034 7198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:39.211735 7198 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:37203
I20260620 20:09:39.211753 7264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:37203 every 8 connection(s)
I20260620 20:09:39.213078 7198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:39.215536 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 7198
I20260620 20:09:39.215734 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:39.216363 4774 external_mini_cluster.cc:1428] Setting key aebfb85be53b0f559e3c000765cb1f5c
I20260620 20:09:39.216923 7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:39.231148 7265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd: Bootstrap starting.
Jun 20 20:09:39 dist-test-slave-4bwq krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:39.234853 7265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:39.236150 7265 log.cc:824] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:39.239596 7265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd: No bootstrap required, opened a new log
I20260620 20:09:39.240513 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:39.218250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.4.169.190:37203)
0620 20:09:39.219096 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:39.219110 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:39.219159 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:09:39.221554 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:39.221591 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:39.222093 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:39.222550 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:39.223804 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:39.225745 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:39.227390 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:39.228505 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:39.228683 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:39.232809 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:39.232820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:39.232833 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:39.232873 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:39.236556 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:39.237246 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:39.237257 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:39.237269 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:39.237519 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:39.237896 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:39.237903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:39.237906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:39.238434 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:39.238720 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:39.239149 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 20:09:39.239548 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":324,"threads_started":1}
I20260620 20:09:39.248076 7265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb453601aeb43ce869db82fcf72f2cd" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37203 } }
I20260620 20:09:39.248500 7265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:39.248603 7265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb453601aeb43ce869db82fcf72f2cd, State: Initialized, Role: FOLLOWER
I20260620 20:09:39.249058 7265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb453601aeb43ce869db82fcf72f2cd" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37203 } }
I20260620 20:09:39.249236 7265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:39.249330 7265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:39.249437 7265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:39.251446 7265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb453601aeb43ce869db82fcf72f2cd" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37203 } }
I20260620 20:09:39.251847 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb453601aeb43ce869db82fcf72f2cd; no voters:
I20260620 20:09:39.252326 7265 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:39.252458 7270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:39.253125 7270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [term 1 LEADER]: Becoming Leader. State: Replica: 5cb453601aeb43ce869db82fcf72f2cd, State: Running, Role: LEADER
I20260620 20:09:39.253774 7270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb453601aeb43ce869db82fcf72f2cd" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37203 } }
I20260620 20:09:39.254242 7265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:39.257121 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb453601aeb43ce869db82fcf72f2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb453601aeb43ce869db82fcf72f2cd" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37203 } } }
I20260620 20:09:39.257524 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:39.257961 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb453601aeb43ce869db82fcf72f2cd. Latest consensus state: current_term: 1 leader_uuid: "5cb453601aeb43ce869db82fcf72f2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb453601aeb43ce869db82fcf72f2cd" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37203 } } }
I20260620 20:09:39.258484 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:39.259114 7279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:39.264034 7279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:39.272199 7279 catalog_manager.cc:1347] Generated new cluster ID: 63fc8fa3384846a4b25c72a86d11cec5
I20260620 20:09:39.272311 7279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:39.284008 7279 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:39.285717 7279 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:39.299523 7279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb453601aeb43ce869db82fcf72f2cd: Generated new TSK 0
I20260620 20:09:39.300595 7279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:09:39.356091 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:37203
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:39.689005 7291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:39.689355 7291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:39.689533 7291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:39.696771 7291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:39.696909 7291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:39.697086 7291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:09:39.706563 7291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:37203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:39.707827 7291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:39.709093 7291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:39.722244 7296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:39.722218 7299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:39.722179 7297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:39.723080 7291 server_base.cc:1034] running on GCE node
I20260620 20:09:39.723743 7291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:39.724728 7291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:39.725965 7291 hybrid_clock.cc:648] HybridClock initialized: now 1781986179725967 us; error 109 us; skew 500 ppm
I20260620 20:09:39.726300 7291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:39 dist-test-slave-4bwq krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986179, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:39.729417 7291 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:09:39.730916 7291 webserver.cc:460] Webserver started at http://127.4.169.129:43943/ using document root <none> and password file <none>
I20260620 20:09:39.731626 7291 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:39.731775 7291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:39.732097 7291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:39.735105 7291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "af0edd2b07c24b7c909b82dcfc135d32"
format_stamp: "Formatted at 2026-06-20 20:09:39 on dist-test-slave-4bwq"
server_key: "ddd1df565612c9c18de2a1a3c3f5df4f"
server_key_iv: "32efc5f07758740bd72c0e26c0ac11bb"
server_key_version: "encryptionkey@0"
I20260620 20:09:39.735874 7291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "af0edd2b07c24b7c909b82dcfc135d32"
format_stamp: "Formatted at 2026-06-20 20:09:39 on dist-test-slave-4bwq"
server_key: "ddd1df565612c9c18de2a1a3c3f5df4f"
server_key_iv: "32efc5f07758740bd72c0e26c0ac11bb"
server_key_version: "encryptionkey@0"
I20260620 20:09:39.741395 7291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:09:39.744860 7306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:39.746098 7291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:09:39.746264 7291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "af0edd2b07c24b7c909b82dcfc135d32"
format_stamp: "Formatted at 2026-06-20 20:09:39 on dist-test-slave-4bwq"
server_key: "ddd1df565612c9c18de2a1a3c3f5df4f"
server_key_iv: "32efc5f07758740bd72c0e26c0ac11bb"
server_key_version: "encryptionkey@0"
I20260620 20:09:39.746567 7291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:39.770795 7291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:39.774864 7291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:39.775147 7291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:39.776054 7291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:39.777304 7291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:39.777377 7291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:39.777496 7291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:39.777556 7291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:39.811348 7291 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:41381
I20260620 20:09:39.811436 7418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:41381 every 8 connection(s)
I20260620 20:09:39.812599 7291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:09:39.815917 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 7291
I20260620 20:09:39.816123 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:09:39.816962 4774 external_mini_cluster.cc:1428] Setting key f7fbf57c7c38e3eba7c88b89e9dff565
Jun 20 20:09:39 dist-test-slave-4bwq krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986179, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:39.835677 7421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:39.815187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:58741 (local address 127.4.169.190:37203)
0620 20:09:39.815705 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:39.815713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:39.816839 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 20:09:39.819094 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:39.819098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:39.819156 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:39.819241 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:39.821337 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:39.822013 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:39.824472 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:39.825084 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:39.825224 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:39.828688 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:39.828697 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:39.828706 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:39.828749 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:39.832308 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:39.833116 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:39.833124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:39.833128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:39.833281 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:39.833701 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:39.833709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:39.833715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:39.834114 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:39.834284 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:39.835021 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 20:09:39.835259 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260620 20:09:39.836308 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:39.815747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:37203 (local address 127.4.169.129:58741)
0620 20:09:39.816611 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:39.816696 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:39.818820 (+ 2124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:39.819509 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:39.819520 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:39.820315 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:39.821164 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:39.821193 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:39.822175 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:39.822183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:39.824336 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:39.824353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:39.825323 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:39.825328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:39.825523 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:39.826222 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:39.826243 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:39.828468 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:39.832495 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:39.832504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:39.832525 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:39.832976 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:39.833410 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:39.833413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:39.833415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:39.833580 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:39.834263 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:39.834269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:39.834620 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 20:09:39.834855 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 20:09:39.835227 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":341,"threads_started":1}
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:09:39.837960 7419 heartbeater.cc:347] Connected to a master server at 127.4.169.190:37203
I20260620 20:09:39.838279 7419 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:39.839092 7419 heartbeater.cc:511] Master 127.4.169.190:37203 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
I20260620 20:09:39.841377 7230 ts_manager.cc:194] Registered new tserver with Master: af0edd2b07c24b7c909b82dcfc135d32 (127.4.169.129:41381)
I20260620 20:09:39.842970 7230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:58741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:09:39.861261 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:37203
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:40.176373 7425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:40.176633 7425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:40.176805 7425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:40.183888 7425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:40.183983 7425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:40.184188 7425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:09:40.193367 7425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:37203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:40.194660 7425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:40.195924 7425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:40.209118 7430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:40.209249 7431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:40.209282 7433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:40.209904 7425 server_base.cc:1034] running on GCE node
I20260620 20:09:40.210556 7425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:40.211509 7425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:40.212736 7425 hybrid_clock.cc:648] HybridClock initialized: now 1781986180212679 us; error 49 us; skew 500 ppm
I20260620 20:09:40.213033 7425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:40 dist-test-slave-4bwq krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986180, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:40.216312 7425 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:09:40.217864 7425 webserver.cc:460] Webserver started at http://127.4.169.130:42109/ using document root <none> and password file <none>
I20260620 20:09:40.218600 7425 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:40.218750 7425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:40.219051 7425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:40.221864 7425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "7f01abf431624a7887258e016c65c8b6"
format_stamp: "Formatted at 2026-06-20 20:09:40 on dist-test-slave-4bwq"
server_key: "55958a30fd2a662a949c00bfd27f03d9"
server_key_iv: "c110b8d77fa79407ddfee8093e8f472f"
server_key_version: "encryptionkey@0"
I20260620 20:09:40.222733 7425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "7f01abf431624a7887258e016c65c8b6"
format_stamp: "Formatted at 2026-06-20 20:09:40 on dist-test-slave-4bwq"
server_key: "55958a30fd2a662a949c00bfd27f03d9"
server_key_iv: "c110b8d77fa79407ddfee8093e8f472f"
server_key_version: "encryptionkey@0"
I20260620 20:09:40.227941 7425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:09:40.231261 7440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:40.232476 7425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:09:40.232657 7425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "7f01abf431624a7887258e016c65c8b6"
format_stamp: "Formatted at 2026-06-20 20:09:40 on dist-test-slave-4bwq"
server_key: "55958a30fd2a662a949c00bfd27f03d9"
server_key_iv: "c110b8d77fa79407ddfee8093e8f472f"
server_key_version: "encryptionkey@0"
I20260620 20:09:40.232864 7425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:40.265291 7425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:40.269311 7425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:40.269598 7425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:40.270704 7425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:40.272106 7425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:40.272208 7425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:40.272363 7425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:40.272436 7425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:40.305886 7425 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:43737
I20260620 20:09:40.305934 7553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:43737 every 8 connection(s)
I20260620 20:09:40.307255 7425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:09:40.308588 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 7425
I20260620 20:09:40.308780 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:09:40.309226 4774 external_mini_cluster.cc:1428] Setting key 7fbfa01ad7004c00beb62a95f85529f3
Jun 20 20:09:40 dist-test-slave-4bwq krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986180, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:09:40.330634 7421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:40.309762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:53693 (local address 127.4.169.190:37203)
0620 20:09:40.309919 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:40.309923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:40.311654 (+ 1731us) server_negotiation.cc:407] Connection header received
0620 20:09:40.314545 (+ 2891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:40.314549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:40.314612 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:40.314701 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:40.316782 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:40.317623 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:40.319469 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:40.320254 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:40.320392 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:40.323929 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:40.323938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:40.323946 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:40.323990 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:40.327665 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:40.328370 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:40.328379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:40.328385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:40.328538 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:40.328926 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:40.328931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:40.328934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:40.329311 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:40.329478 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:40.330092 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 20:09:40.330337 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
I20260620 20:09:40.331740 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:40.310465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:37203 (local address 127.4.169.130:53693)
0620 20:09:40.311488 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:40.311537 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:40.314227 (+ 2690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:40.314977 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:40.314991 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:40.315791 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:40.316605 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:40.316630 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:40.317782 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:40.317788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:40.319329 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:40.319341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:40.320488 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:40.320493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:40.320695 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:40.321383 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:40.321404 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:40.323703 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:40.327817 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:40.327824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:40.327836 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:40.328230 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:40.328646 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:40.328649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:40.328651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:40.328810 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:40.329462 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:40.329468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:40.329773 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 20:09:40.329990 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 20:09:40.330479 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":350,"threads_started":1}
I20260620 20:09:40.333194 7554 heartbeater.cc:347] Connected to a master server at 127.4.169.190:37203
I20260620 20:09:40.333515 7554 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:40.334368 7554 heartbeater.cc:511] Master 127.4.169.190:37203 requested a full tablet report, sending...
I20260620 20:09:40.335955 7230 ts_manager.cc:194] Registered new tserver with Master: 7f01abf431624a7887258e016c65c8b6 (127.4.169.130:43737)
I20260620 20:09:40.337056 7230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:53693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:09:40.355083 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:37203
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:40.676522 7564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:40.676777 7564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:40.676930 7564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:40.683997 7564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:40.684094 7564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:40.684309 7564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:09:40.693709 7564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:37203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:40.694900 7564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:40.696170 7564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:40.709038 7569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:40.709012 7570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:40.709092 7572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:40.710037 7564 server_base.cc:1034] running on GCE node
I20260620 20:09:40.710716 7564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:40.711812 7564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:40.713074 7564 hybrid_clock.cc:648] HybridClock initialized: now 1781986180713030 us; error 33 us; skew 500 ppm
I20260620 20:09:40.713413 7564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:40 dist-test-slave-4bwq krb5kdc[7185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986180, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:40.716527 7564 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:09:40.718009 7564 webserver.cc:460] Webserver started at http://127.4.169.131:38843/ using document root <none> and password file <none>
I20260620 20:09:40.718779 7564 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:40.718922 7564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:40.719219 7564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:40.722050 7564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "e3f030af5be54facbcd6b73dad58abd7"
format_stamp: "Formatted at 2026-06-20 20:09:40 on dist-test-slave-4bwq"
server_key: "b16389f0ca20e471d98261a435a841c5"
server_key_iv: "ef2c59af851f049b0dce5a61c531414f"
server_key_version: "encryptionkey@0"
I20260620 20:09:40.722960 7564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "e3f030af5be54facbcd6b73dad58abd7"
format_stamp: "Formatted at 2026-06-20 20:09:40 on dist-test-slave-4bwq"
server_key: "b16389f0ca20e471d98261a435a841c5"
server_key_iv: "ef2c59af851f049b0dce5a61c531414f"
server_key_version: "encryptionkey@0"
I20260620 20:09:40.728394 7564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:09:40.731945 7579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:40.733361 7564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:09:40.733490 7564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "e3f030af5be54facbcd6b73dad58abd7"
format_stamp: "Formatted at 2026-06-20 20:09:40 on dist-test-slave-4bwq"
server_key: "b16389f0ca20e471d98261a435a841c5"
server_key_iv: "ef2c59af851f049b0dce5a61c531414f"
server_key_version: "encryptionkey@0"
I20260620 20:09:40.733651 7564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:40.749909 7564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:40.754364 7564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:40.754701 7564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:40.755689 7564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:40.756970 7564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:40.757040 7564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:40.757205 7564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:40.757258 7564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:40.789976 7564 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:36453
I20260620 20:09:40.790014 7691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:36453 every 8 connection(s)
I20260620 20:09:40.791244 7564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:09:40.793452 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 7564
I20260620 20:09:40.793606 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:09:40.794152 4774 external_mini_cluster.cc:1428] Setting key 9b49a3dae00ace5bf3a84b8e1f826bef
Jun 20 20:09:40 dist-test-slave-4bwq krb5kdc[7185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986180, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:40.810910 7421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:40.793857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:38293 (local address 127.4.169.190:37203)
0620 20:09:40.794070 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:40.794077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:40.795420 (+ 1343us) server_negotiation.cc:407] Connection header received
0620 20:09:40.797215 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:40.797219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:40.797286 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:40.797365 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:40.798943 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:40.799676 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:40.801090 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:40.801474 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:40.801564 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:40.804783 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:40.804792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:40.804799 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:40.804840 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:40.807896 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:40.808715 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:40.808721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:40.808725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:40.808917 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:40.809423 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:40.809427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:40.809429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:40.809677 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:40.809782 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:40.810472 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 20:09:40.810624 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:09:40.812158 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:40.794466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:37203 (local address 127.4.169.131:38293)
0620 20:09:40.795263 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:40.795314 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:40.796987 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:40.797569 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:40.797576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:40.798178 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:40.798802 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:40.798819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:40.799810 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:40.799813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:40.800975 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:40.800986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:40.801642 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:40.801645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:40.801846 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:40.802524 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:40.802545 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:40.804541 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:40.808056 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:40.808064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:40.808078 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:40.808596 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:40.809043 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:40.809048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:40.809052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:40.809316 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:40.809793 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:40.809798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:40.810102 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 20:09:40.810315 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 20:09:40.810861 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":332,"threads_started":1}
I20260620 20:09:40.813532 7692 heartbeater.cc:347] Connected to a master server at 127.4.169.190:37203
I20260620 20:09:40.813848 7692 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:40.814726 7692 heartbeater.cc:511] Master 127.4.169.190:37203 requested a full tablet report, sending...
I20260620 20:09:40.816179 7230 ts_manager.cc:194] Registered new tserver with Master: e3f030af5be54facbcd6b73dad58abd7 (127.4.169.131:36453)
I20260620 20:09:40.816952 7230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:38293
I20260620 20:09:40.821026 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:09:40.845971 7419 heartbeater.cc:503] Master 127.4.169.190:37203 was elected leader, sending a full tablet report...
W20260620 20:09:40.851037 7421 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:09:40.848534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.4.169.190:37203)
0620 20:09:40.848767 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:40.848777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:40.849148 (+ 371us) server_negotiation.cc:407] Connection header received
0620 20:09:40.849312 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:40.849316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:40.849411 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:40.849504 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:40.850758 (+ 1254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36344: BlockingRecv error: recv got EOF from 127.0.0.1:36344 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:09:40.853914 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7291
W20260620 20:09:40.856911 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7291: Not authorized: Client connection negotiation failed: client connection to 127.4.169.129:41381: server requires authentication, but client does not have Kerberos credentials available
I20260620 20:09:40.857111 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 7291
I20260620 20:09:40.873587 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7425
W20260620 20:09:40.876317 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7425: Not authorized: Client connection negotiation failed: client connection to 127.4.169.130:43737: server requires authentication, but client does not have Kerberos credentials available
I20260620 20:09:40.876464 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 7425
I20260620 20:09:40.891146 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7564
W20260620 20:09:40.893646 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7564: Not authorized: Client connection negotiation failed: client connection to 127.4.169.131:36453: server requires authentication, but client does not have Kerberos credentials available
I20260620 20:09:40.893802 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 7564
I20260620 20:09:40.908254 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7198
I20260620 20:09:40.909310 7260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:41.010927 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 7198
2026-06-20T20:09:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3492 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:09:41 dist-test-slave-4bwq krb5kdc[7709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:09:41 dist-test-slave-4bwq krb5kdc[7709](info): set up 2 sockets
Jun 20 20:09:41 dist-test-slave-4bwq krb5kdc[7709](info): commencing operation
krb5kdc: starting...
W20260620 20:09:42.103693 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:09:42 dist-test-slave-4bwq krb5kdc[7709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:09:42.289330 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:37651
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:37651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:42.607161 7722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:42.607426 7722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:42.607618 7722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:42.614993 7722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:42.615088 7722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:42.615226 7722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:42.615309 7722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:42.615367 7722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:42.624804 7722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:37651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:37651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:42.625996 7722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:42.627323 7722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:42.638913 7727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:42.638942 7728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:42.639195 7730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:42.639539 7722 server_base.cc:1034] running on GCE node
I20260620 20:09:42.640309 7722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:42.641628 7722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:42.642863 7722 hybrid_clock.cc:648] HybridClock initialized: now 1781986182642809 us; error 69 us; skew 500 ppm
I20260620 20:09:42.643196 7722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:42 dist-test-slave-4bwq krb5kdc[7709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986182, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:42.650643 7722 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:09:42.652002 7722 webserver.cc:460] Webserver started at http://127.4.169.190:36239/ using document root <none> and password file <none>
I20260620 20:09:42.652626 7722 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:42.652720 7722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:42.653023 7722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:42.656018 7722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "3f901a63a9124a46bf144f670a33755c"
format_stamp: "Formatted at 2026-06-20 20:09:42 on dist-test-slave-4bwq"
server_key: "63d13948201b13e8e18751e017463bbe"
server_key_iv: "192be96a8b1471375cc3e8c7de068acc"
server_key_version: "encryptionkey@0"
I20260620 20:09:42.656773 7722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "3f901a63a9124a46bf144f670a33755c"
format_stamp: "Formatted at 2026-06-20 20:09:42 on dist-test-slave-4bwq"
server_key: "63d13948201b13e8e18751e017463bbe"
server_key_iv: "192be96a8b1471375cc3e8c7de068acc"
server_key_version: "encryptionkey@0"
I20260620 20:09:42.661803 7722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:09:42.665151 7737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:42.666514 7722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:09:42.666688 7722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "3f901a63a9124a46bf144f670a33755c"
format_stamp: "Formatted at 2026-06-20 20:09:42 on dist-test-slave-4bwq"
server_key: "63d13948201b13e8e18751e017463bbe"
server_key_iv: "192be96a8b1471375cc3e8c7de068acc"
server_key_version: "encryptionkey@0"
I20260620 20:09:42.666939 7722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:42.685989 7722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:42.689989 7722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:42.690275 7722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:42.708257 7722 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:37651
I20260620 20:09:42.708292 7788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:37651 every 8 connection(s)
I20260620 20:09:42.709561 7722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:42.713003 7789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:42.716315 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 7722
I20260620 20:09:42.716529 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:42.717274 4774 external_mini_cluster.cc:1428] Setting key 49fb13620a3139c2cbad7bca3d6c1194
I20260620 20:09:42.724812 7789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c: Bootstrap starting.
I20260620 20:09:42.727892 7789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:42.728871 7789 log.cc:824] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:09:42 dist-test-slave-4bwq krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:42.731494 7789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c: No bootstrap required, opened a new log
I20260620 20:09:42.738269 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:42.719204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.4.169.190:37651)
0620 20:09:42.719967 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:42.719979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:42.720016 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:09:42.721580 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:42.721600 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:42.721989 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:42.722373 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:42.723771 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:42.725115 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:42.726343 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:42.727050 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:42.727245 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:42.731066 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:42.731073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:42.731087 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:42.731120 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:42.734353 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:42.735061 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:42.735067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:42.735073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:42.735221 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:42.735594 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:42.735598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:42.735603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:42.736133 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:42.736446 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:42.736922 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 20:09:42.737306 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":346,"threads_started":1}
I20260620 20:09:42.739446 7789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f901a63a9124a46bf144f670a33755c" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37651 } }
I20260620 20:09:42.739899 7789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:42.740015 7789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f901a63a9124a46bf144f670a33755c, State: Initialized, Role: FOLLOWER
I20260620 20:09:42.740614 7789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f901a63a9124a46bf144f670a33755c" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37651 } }
I20260620 20:09:42.740885 7789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:42.741031 7789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:42.741216 7789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:42.742158 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:09:42.743672 7789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f901a63a9124a46bf144f670a33755c" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37651 } }
I20260620 20:09:42.744155 7789 leader_election.cc:302] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f901a63a9124a46bf144f670a33755c; no voters:
I20260620 20:09:42.744763 7789 leader_election.cc:288] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:42.744879 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:42.746635 7794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [term 1 LEADER]: Becoming Leader. State: Replica: 3f901a63a9124a46bf144f670a33755c, State: Running, Role: LEADER
I20260620 20:09:42.747285 7789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:42.747174 7794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f901a63a9124a46bf144f670a33755c" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37651 } }
I20260620 20:09:42.750022 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f901a63a9124a46bf144f670a33755c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f901a63a9124a46bf144f670a33755c" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37651 } } }
I20260620 20:09:42.750489 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:42.750125 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f901a63a9124a46bf144f670a33755c. Latest consensus state: current_term: 1 leader_uuid: "3f901a63a9124a46bf144f670a33755c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f901a63a9124a46bf144f670a33755c" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37651 } } }
I20260620 20:09:42.750702 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:42.762372 7801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:42.768146 7801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:42.788708 7801 catalog_manager.cc:1347] Generated new cluster ID: b114d62e26bb425eae7bfb18088b5809
I20260620 20:09:42.788844 7801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:42.797394 7816 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:09:42.809652 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:42.799773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.4.169.190:37651)
0620 20:09:42.799999 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:42.800008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:42.803057 (+ 3049us) server_negotiation.cc:407] Connection header received
0620 20:09:42.803283 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:42.803291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:42.803364 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:42.803484 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:42.804490 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:42.805203 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:42.806509 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:42.806941 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:42.807077 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:42.808180 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:42.808185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:42.808188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:42.808848 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:42.808905 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:42.808932 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:42.809058 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:42.809195 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:09:42.809291 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:09:42.844673 7801 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:42.846122 7801 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:42.856310 7801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f901a63a9124a46bf144f670a33755c: Generated new TSK 0
I20260620 20:09:42.857272 7801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:09:42.871548 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7722
I20260620 20:09:42.872712 7784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:42.965471 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 7722
2026-06-20T20:09:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1948 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:09:43 dist-test-slave-4bwq krb5kdc[7823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:09:43 dist-test-slave-4bwq krb5kdc[7823](info): set up 2 sockets
Jun 20 20:09:43 dist-test-slave-4bwq krb5kdc[7823](info): commencing operation
krb5kdc: starting...
W20260620 20:09:44.035058 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:09:44 dist-test-slave-4bwq krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:09:44.239890 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:36869
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:36869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:44.569756 7836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:44.570045 7836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:44.570223 7836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:44.577698 7836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:44.577813 7836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:44.577924 7836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:44.578025 7836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:44.578104 7836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:44.588227 7836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:36869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:36869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:44.589565 7836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:44.590926 7836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:44.602670 7842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:44.602849 7841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:44.602890 7844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:44.603718 7836 server_base.cc:1034] running on GCE node
I20260620 20:09:44.604487 7836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:44.605782 7836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:44.606972 7836 hybrid_clock.cc:648] HybridClock initialized: now 1781986184606908 us; error 50 us; skew 500 ppm
I20260620 20:09:44.607304 7836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:44 dist-test-slave-4bwq krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986184, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:44.610351 7836 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:09:44.611878 7836 webserver.cc:460] Webserver started at http://127.4.169.190:39777/ using document root <none> and password file <none>
I20260620 20:09:44.612504 7836 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:44.612603 7836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:44.612839 7836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:44.615747 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "db4a84ec1d7a43f1b64ba78818b605e0"
format_stamp: "Formatted at 2026-06-20 20:09:44 on dist-test-slave-4bwq"
server_key: "38fae4577b825a154ee5520395bc626b"
server_key_iv: "45cfd9d7ad36df780ba5d73990c81f26"
server_key_version: "encryptionkey@0"
I20260620 20:09:44.616431 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "db4a84ec1d7a43f1b64ba78818b605e0"
format_stamp: "Formatted at 2026-06-20 20:09:44 on dist-test-slave-4bwq"
server_key: "38fae4577b825a154ee5520395bc626b"
server_key_iv: "45cfd9d7ad36df780ba5d73990c81f26"
server_key_version: "encryptionkey@0"
I20260620 20:09:44.621445 7836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:09:44.624953 7851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:44.626367 7836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:09:44.626638 7836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "db4a84ec1d7a43f1b64ba78818b605e0"
format_stamp: "Formatted at 2026-06-20 20:09:44 on dist-test-slave-4bwq"
server_key: "38fae4577b825a154ee5520395bc626b"
server_key_iv: "45cfd9d7ad36df780ba5d73990c81f26"
server_key_version: "encryptionkey@0"
I20260620 20:09:44.626864 7836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:44.641973 7836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:44.646134 7836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:44.646464 7836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:44.667451 7836 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:36869
I20260620 20:09:44.667475 7902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:36869 every 8 connection(s)
I20260620 20:09:44.668829 7836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:44.672523 7903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:44.678261 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 7836
I20260620 20:09:44.678516 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:44.679098 4774 external_mini_cluster.cc:1428] Setting key 12d0ce7d51a8703f64cf7829bf964841
I20260620 20:09:44.682981 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0: Bootstrap starting.
I20260620 20:09:44.686754 7903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:44.688006 7903 log.cc:824] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:44.691395 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0: No bootstrap required, opened a new log
Jun 20 20:09:44 dist-test-slave-4bwq krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:44.700378 7903 raft_consensus.cc:348] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a84ec1d7a43f1b64ba78818b605e0" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36869 } }
I20260620 20:09:44.700851 7903 raft_consensus.cc:374] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:44.700984 7903 raft_consensus.cc:729] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4a84ec1d7a43f1b64ba78818b605e0, State: Initialized, Role: FOLLOWER
I20260620 20:09:44.701671 7903 consensus_queue.cc:260] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a84ec1d7a43f1b64ba78818b605e0" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36869 } }
I20260620 20:09:44.701954 7903 raft_consensus.cc:388] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:44.702107 7903 raft_consensus.cc:482] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:44.702327 7903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:44.704603 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:44.681461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.4.169.190:36869)
0620 20:09:44.682299 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:44.682312 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:44.682354 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:09:44.684340 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:44.684367 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:44.684862 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:44.685231 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:44.687073 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:44.688644 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:44.690366 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:44.691132 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:44.691285 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:44.696056 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:44.696065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:44.696083 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:44.696132 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:44.699692 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:44.700533 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:44.700541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:44.700549 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:44.700740 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:44.701299 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:44.701303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:44.701305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:44.702121 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:44.702505 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:44.703008 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 20:09:44.703503 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":396,"threads_started":1}
I20260620 20:09:44.705039 7903 raft_consensus.cc:504] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a84ec1d7a43f1b64ba78818b605e0" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36869 } }
I20260620 20:09:44.705576 7903 leader_election.cc:302] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db4a84ec1d7a43f1b64ba78818b605e0; no voters:
I20260620 20:09:44.706306 7903 leader_election.cc:288] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:44.706452 7908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:44.709134 7903 sys_catalog.cc:564] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:44.710005 7908 raft_consensus.cc:686] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [term 1 LEADER]: Becoming Leader. State: Replica: db4a84ec1d7a43f1b64ba78818b605e0, State: Running, Role: LEADER
I20260620 20:09:44.710807 7908 consensus_queue.cc:237] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a84ec1d7a43f1b64ba78818b605e0" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36869 } }
I20260620 20:09:44.714103 7910 sys_catalog.cc:455] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db4a84ec1d7a43f1b64ba78818b605e0. Latest consensus state: current_term: 1 leader_uuid: "db4a84ec1d7a43f1b64ba78818b605e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a84ec1d7a43f1b64ba78818b605e0" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36869 } } }
I20260620 20:09:44.714515 7910 sys_catalog.cc:458] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:44.715291 7909 sys_catalog.cc:455] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db4a84ec1d7a43f1b64ba78818b605e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a84ec1d7a43f1b64ba78818b605e0" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 36869 } } }
I20260620 20:09:44.715678 7909 sys_catalog.cc:458] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:44.718446 7918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:44.724198 7918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:44.733801 7918 catalog_manager.cc:1347] Generated new cluster ID: fdd6009724e442acafb80a81cf41361c
I20260620 20:09:44.733930 7918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:44.758831 7918 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:44.760947 7918 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:44.775877 7918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db4a84ec1d7a43f1b64ba78818b605e0: Generated new TSK 0
I20260620 20:09:44.776958 7918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:09:44.829378 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:36869
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:45.158567 7929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:45.158861 7929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:45.159025 7929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:45.166096 7929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:45.166196 7929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:45.166375 7929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:09:45.175940 7929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:36869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:45.177248 7929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:45.178580 7929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:45.191691 7934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:45.191632 7937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:45.191660 7935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:45.192739 7929 server_base.cc:1034] running on GCE node
I20260620 20:09:45.193354 7929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:45.194300 7929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:45.195505 7929 hybrid_clock.cc:648] HybridClock initialized: now 1781986185195463 us; error 29 us; skew 500 ppm
I20260620 20:09:45.195822 7929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:45 dist-test-slave-4bwq krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986185, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:45.199107 7929 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:09:45.200520 7929 webserver.cc:460] Webserver started at http://127.4.169.129:38499/ using document root <none> and password file <none>
I20260620 20:09:45.201171 7929 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:45.201270 7929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:45.201510 7929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:45.204504 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "74f2c445c5b641bca71c7c93340fbf3f"
format_stamp: "Formatted at 2026-06-20 20:09:45 on dist-test-slave-4bwq"
server_key: "dde957007e9e62fae171e23c4856078a"
server_key_iv: "4ae8fe9d63dedec7026ade7ad0de9a92"
server_key_version: "encryptionkey@0"
I20260620 20:09:45.205296 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "74f2c445c5b641bca71c7c93340fbf3f"
format_stamp: "Formatted at 2026-06-20 20:09:45 on dist-test-slave-4bwq"
server_key: "dde957007e9e62fae171e23c4856078a"
server_key_iv: "4ae8fe9d63dedec7026ade7ad0de9a92"
server_key_version: "encryptionkey@0"
I20260620 20:09:45.210721 7929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:09:45.214566 7944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:45.215880 7929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:45.216064 7929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "74f2c445c5b641bca71c7c93340fbf3f"
format_stamp: "Formatted at 2026-06-20 20:09:45 on dist-test-slave-4bwq"
server_key: "dde957007e9e62fae171e23c4856078a"
server_key_iv: "4ae8fe9d63dedec7026ade7ad0de9a92"
server_key_version: "encryptionkey@0"
I20260620 20:09:45.216282 7929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:45.229925 7929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:45.234146 7929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:45.234511 7929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:45.235442 7929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:45.236763 7929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:45.236869 7929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:45.237020 7929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:45.237097 7929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:45.271157 7929 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:33037
I20260620 20:09:45.271190 8056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:33037 every 8 connection(s)
I20260620 20:09:45.272480 7929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:09:45.279122 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 7929
I20260620 20:09:45.279345 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:09:45.279937 4774 external_mini_cluster.cc:1428] Setting key f7c37d2a54b448d0cb5bc816627c2da0
Jun 20 20:09:45 dist-test-slave-4bwq krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986185, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:45.293562 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:45.275067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:41213 (local address 127.4.169.190:36869)
0620 20:09:45.275537 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:45.275548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:45.276434 (+ 886us) server_negotiation.cc:407] Connection header received
0620 20:09:45.278164 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:45.278168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:45.278230 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:45.278329 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:45.280175 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:45.281104 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:45.282985 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:45.283760 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:45.283907 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:45.288042 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:45.288047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:45.288050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:45.288080 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:45.290626 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:45.291330 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:45.291341 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:45.291343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:45.291458 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:45.291882 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:45.291886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:45.291890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:45.292153 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:45.292303 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:45.292977 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 20:09:45.293188 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":197,"threads_started":1}
I20260620 20:09:45.294726 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:45.275580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36869 (local address 127.4.169.129:41213)
0620 20:09:45.276287 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:45.276324 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:45.277923 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:45.278762 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:45.278772 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:45.279362 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:45.279975 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:45.279993 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:45.281255 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:45.281260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:45.282828 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:45.282841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:45.284004 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:45.284010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:45.284277 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:45.285051 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:45.285083 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:45.287821 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:45.290767 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:45.290775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:45.290793 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:45.291199 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:45.291565 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:45.291571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:45.291575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:45.291766 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:45.292277 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:45.292293 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:45.292627 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 20:09:45.292880 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 20:09:45.293430 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":266,"threads_started":1}
I20260620 20:09:45.296442 8057 heartbeater.cc:347] Connected to a master server at 127.4.169.190:36869
I20260620 20:09:45.296746 8057 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:45.297487 8057 heartbeater.cc:511] Master 127.4.169.190:36869 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:09:45.299664 7868 ts_manager.cc:194] Registered new tserver with Master: 74f2c445c5b641bca71c7c93340fbf3f (127.4.169.129:33037)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
I20260620 20:09:45.301107 7868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:41213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:09:45.325582 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:36869
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:45.654397 8063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:45.654750 8063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:45.654901 8063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:45.662175 8063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:45.662297 8063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:45.662518 8063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:09:45.672420 8063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:36869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:45.673733 8063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:45.675083 8063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:45.688238 8068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:45.688215 8071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:45.688256 8069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:45.691994 8063 server_base.cc:1034] running on GCE node
I20260620 20:09:45.692646 8063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:45.693660 8063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:45.694859 8063 hybrid_clock.cc:648] HybridClock initialized: now 1781986185694809 us; error 39 us; skew 500 ppm
I20260620 20:09:45.695186 8063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:45 dist-test-slave-4bwq krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986185, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:45.698488 8063 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:09:45.700040 8063 webserver.cc:460] Webserver started at http://127.4.169.130:36865/ using document root <none> and password file <none>
I20260620 20:09:45.700767 8063 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:45.700875 8063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:45.701213 8063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:45.704207 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "1ecd9bf8aeb24f71895ff8eb673a004b"
format_stamp: "Formatted at 2026-06-20 20:09:45 on dist-test-slave-4bwq"
server_key: "21424654fb8851e5d5b2e4ccd1552f13"
server_key_iv: "bae62b61a04967aee653c8c5fc2a89ea"
server_key_version: "encryptionkey@0"
I20260620 20:09:45.705001 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "1ecd9bf8aeb24f71895ff8eb673a004b"
format_stamp: "Formatted at 2026-06-20 20:09:45 on dist-test-slave-4bwq"
server_key: "21424654fb8851e5d5b2e4ccd1552f13"
server_key_iv: "bae62b61a04967aee653c8c5fc2a89ea"
server_key_version: "encryptionkey@0"
I20260620 20:09:45.710364 8063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:09:45.713917 8078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:45.715447 8063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:09:45.715624 8063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "1ecd9bf8aeb24f71895ff8eb673a004b"
format_stamp: "Formatted at 2026-06-20 20:09:45 on dist-test-slave-4bwq"
server_key: "21424654fb8851e5d5b2e4ccd1552f13"
server_key_iv: "bae62b61a04967aee653c8c5fc2a89ea"
server_key_version: "encryptionkey@0"
I20260620 20:09:45.715829 8063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:45.735754 8063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:45.740232 8063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:45.740527 8063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:45.741439 8063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:45.742873 8063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:45.742975 8063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:45.743127 8063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:45.743201 8063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:45.776633 8063 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:34715
I20260620 20:09:45.776690 8190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:34715 every 8 connection(s)
I20260620 20:09:45.777897 8063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:09:45.784941 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 8063
I20260620 20:09:45.785161 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:09:45.785725 4774 external_mini_cluster.cc:1428] Setting key 0b686c7ed1a27bcfff98cee6fb7f0539
Jun 20 20:09:45 dist-test-slave-4bwq krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986185, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:45.798703 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:45.780403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:44717 (local address 127.4.169.190:36869)
0620 20:09:45.780572 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:45.780576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:45.781791 (+ 1215us) server_negotiation.cc:407] Connection header received
0620 20:09:45.783642 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:45.783648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:45.783734 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:45.783842 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:45.785593 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:45.786440 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:45.788492 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:45.788895 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:45.788986 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:45.793061 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:45.793066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:45.793069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:45.793100 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:45.795794 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:45.796472 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:45.796480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:45.796486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:45.796614 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:45.797027 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:45.797034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:45.797038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:45.797297 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:45.797453 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:45.798099 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 20:09:45.798331 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:09:45.799680 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:45.780912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36869 (local address 127.4.169.130:44717)
0620 20:09:45.781606 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:45.781663 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:45.783373 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:45.784075 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:45.784083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:45.784604 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:45.785419 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:45.785443 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:45.786604 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:45.786609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:45.788333 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:45.788347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:45.789120 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:45.789127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:45.789459 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:45.790187 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:45.790228 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:45.792841 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:45.795924 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:45.795933 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:45.795950 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:45.796354 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:45.796716 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:45.796722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:45.796726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:45.796916 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:45.797428 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:45.797436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:45.797754 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 20:09:45.798035 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 20:09:45.798582 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":280,"threads_started":1}
I20260620 20:09:45.801143 8191 heartbeater.cc:347] Connected to a master server at 127.4.169.190:36869
I20260620 20:09:45.801481 8191 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:45.802443 8191 heartbeater.cc:511] Master 127.4.169.190:36869 requested a full tablet report, sending...
I20260620 20:09:45.804075 7868 ts_manager.cc:194] Registered new tserver with Master: 1ecd9bf8aeb24f71895ff8eb673a004b (127.4.169.130:34715)
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:09:45.805179 7868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:44717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:09:45.829272 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:36869
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:46.144132 8196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:46.144429 8196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:46.144523 8196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:46.151757 8196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:46.151867 8196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:46.151978 8196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:09:46.161525 8196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:36869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:46.162918 8196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:46.164167 8196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:46.176702 8202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:46.176826 8201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:46.176882 8204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:46.177896 8196 server_base.cc:1034] running on GCE node
I20260620 20:09:46.178522 8196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:46.179414 8196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:46.180609 8196 hybrid_clock.cc:648] HybridClock initialized: now 1781986186180572 us; error 25 us; skew 500 ppm
I20260620 20:09:46.180915 8196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986186, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:46.183962 8196 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:09:46.185352 8196 webserver.cc:460] Webserver started at http://127.4.169.131:37229/ using document root <none> and password file <none>
I20260620 20:09:46.185988 8196 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:46.186115 8196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:46.186491 8196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:46.189254 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "601f12b2fc124a65b54f8c793aa948a5"
format_stamp: "Formatted at 2026-06-20 20:09:46 on dist-test-slave-4bwq"
server_key: "390b691961912b1d72f0925154268413"
server_key_iv: "b30c6d7595e42dccd032ae97258b9255"
server_key_version: "encryptionkey@0"
I20260620 20:09:46.190004 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "601f12b2fc124a65b54f8c793aa948a5"
format_stamp: "Formatted at 2026-06-20 20:09:46 on dist-test-slave-4bwq"
server_key: "390b691961912b1d72f0925154268413"
server_key_iv: "b30c6d7595e42dccd032ae97258b9255"
server_key_version: "encryptionkey@0"
I20260620 20:09:46.194783 8196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260620 20:09:46.197912 8211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:46.199190 8196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:09:46.199354 8196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "601f12b2fc124a65b54f8c793aa948a5"
format_stamp: "Formatted at 2026-06-20 20:09:46 on dist-test-slave-4bwq"
server_key: "390b691961912b1d72f0925154268413"
server_key_iv: "b30c6d7595e42dccd032ae97258b9255"
server_key_version: "encryptionkey@0"
I20260620 20:09:46.199528 8196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:46.211258 8196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:46.215420 8196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:46.215739 8196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:46.216657 8196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:46.217933 8196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:46.218005 8196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:46.218163 8196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:46.218215 8196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:46.249475 8196 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:45579
I20260620 20:09:46.249512 8323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:45579 every 8 connection(s)
I20260620 20:09:46.250771 8196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:09:46.256872 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 8196
I20260620 20:09:46.257097 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:09:46.257555 4774 external_mini_cluster.cc:1428] Setting key 132143334bbb013758dab87b7e0cae39
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986186, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:46.270334 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:46.253286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:52323 (local address 127.4.169.190:36869)
0620 20:09:46.253500 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:46.253507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:46.254721 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 20:09:46.256488 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:46.256494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:46.256592 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:46.256725 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:46.258308 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.259174 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.260662 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.261067 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.261225 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:46.264721 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:46.264726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:46.264730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:46.264764 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:46.267260 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.268005 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.268011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.268014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.268145 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.268591 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.268597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.268601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.268912 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:46.269065 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:46.269746 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 20:09:46.269964 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:09:46.271564 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:46.253847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:36869 (local address 127.4.169.131:52323)
0620 20:09:46.254570 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:46.254608 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:46.256172 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:46.256925 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:46.256933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:46.257514 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:46.258127 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:46.258144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.259321 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.259330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:46.260524 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:46.260535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.261323 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.261330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:46.261565 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:46.262204 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:46.262226 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:46.264471 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:46.267420 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:46.267429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:46.267455 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:46.267879 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:46.268271 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:46.268276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:46.268280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:46.268474 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:46.269037 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:46.269045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:46.269406 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 20:09:46.269680 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 20:09:46.270235 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":263,"threads_started":1}
I20260620 20:09:46.273334 8324 heartbeater.cc:347] Connected to a master server at 127.4.169.190:36869
I20260620 20:09:46.273653 8324 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:46.274535 8324 heartbeater.cc:511] Master 127.4.169.190:36869 requested a full tablet report, sending...
I20260620 20:09:46.275978 7868 ts_manager.cc:194] Registered new tserver with Master: 601f12b2fc124a65b54f8c793aa948a5 (127.4.169.131:45579)
I20260620 20:09:46.276767 7868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:52323
I20260620 20:09:46.285096 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:09:46.304013 8057 heartbeater.cc:503] Master 127.4.169.190:36869 was elected leader, sending a full tablet report...
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[7823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:46.332326 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:46.318538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.4.169.190:36869)
0620 20:09:46.318714 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:46.318718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:46.319204 (+ 486us) server_negotiation.cc:407] Connection header received
0620 20:09:46.319443 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:46.319447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:46.319509 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:46.319614 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:46.321075 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.321795 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.323082 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.323507 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.323640 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:46.327474 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:46.327479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:46.327482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:46.327514 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:46.329835 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.330465 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.330472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.330480 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.330604 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.331008 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.331012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.331014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.331408 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:46.331610 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:46.331896 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 20:09:46.332076 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 20:09:46.332870 7868 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58806
I20260620 20:09:46.334553 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7929
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:46.350449 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:46.335764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.4.169.129:33037)
0620 20:09:46.336342 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:46.336348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:46.336370 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:09:46.336421 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:46.336427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:46.336624 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:46.336788 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:46.337861 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.338904 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.340057 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.340660 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.340791 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:46.344510 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:46.344521 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:46.344536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:46.344575 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:46.347807 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.348443 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.348451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.348454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.348593 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.348911 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.348914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.348917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.349307 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:46.349556 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:46.349752 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 20:09:46.349999 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":281,"threads_started":1}
W20260620 20:09:46.351751 8052 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44994
W20260620 20:09:46.352968 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7929: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:09:46.353178 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 7929
I20260620 20:09:46.368348 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8063
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:09:46.383388 8336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:46.369141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.4.169.130:34715)
0620 20:09:46.369722 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:46.369729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:46.369753 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:09:46.369817 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:46.369823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:46.370024 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:46.370152 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:46.371358 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.372348 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.373503 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.374026 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.374116 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:46.378135 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:46.378146 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:46.378160 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:46.378195 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:46.380760 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.381421 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.381428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.381430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.381548 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.381884 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.381887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.381889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.382192 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:46.382472 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:46.382793 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 20:09:46.382981 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":316,"threads_started":1}
W20260620 20:09:46.384711 8186 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53230
W20260620 20:09:46.385820 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8063: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:09:46.386021 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 8063
I20260620 20:09:46.401458 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8196
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[7823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986186, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:09:46.417594 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:46.402532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.4.169.131:45579)
0620 20:09:46.403168 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:46.403174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:46.403202 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:09:46.403259 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:46.403265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:46.403476 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:46.403593 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:46.405051 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.406086 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.407376 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:46.407933 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:46.408030 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:46.412376 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:46.412388 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:46.412404 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:46.412440 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:46.415289 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.415899 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.415905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.415908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.416020 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:46.416316 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:46.416319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:46.416321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:46.416647 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:46.416880 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:46.417120 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 20:09:46.417314 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":334,"threads_started":1}
W20260620 20:09:46.418996 8319 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48690
W20260620 20:09:46.420023 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8196: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:09:46.420197 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 8196
I20260620 20:09:46.435194 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 7836
I20260620 20:09:46.436241 7898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:46.539508 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 7836
2026-06-20T20:09:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3579 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[8341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[8341](info): set up 2 sockets
Jun 20 20:09:46 dist-test-slave-4bwq krb5kdc[8341](info): commencing operation
krb5kdc: starting...
W20260620 20:09:47.659091 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.008s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:09:47 dist-test-slave-4bwq krb5kdc[8341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:09:47.845368 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34767
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:48.162104 8354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:48.162501 8354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:48.162669 8354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:48.169907 8354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:48.170054 8354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:48.170179 8354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:48.170286 8354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:48.170373 8354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:48.180059 8354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:48.181367 8354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:48.182730 8354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:48.194365 8359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:48.198884 8360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:48.198879 8362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:48.198925 8354 server_base.cc:1034] running on GCE node
I20260620 20:09:48.199843 8354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:48.201216 8354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:48.202508 8354 hybrid_clock.cc:648] HybridClock initialized: now 1781986188202463 us; error 31 us; skew 500 ppm
I20260620 20:09:48.202862 8354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:48 dist-test-slave-4bwq krb5kdc[8341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986188, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:48.210793 8354 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:09:48.212327 8354 webserver.cc:460] Webserver started at http://127.4.169.190:41273/ using document root <none> and password file <none>
I20260620 20:09:48.213042 8354 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:48.213193 8354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:48.213521 8354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:48.216486 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "2b9bf26fb26c4461a7dd8e9412c495b5"
format_stamp: "Formatted at 2026-06-20 20:09:48 on dist-test-slave-4bwq"
server_key: "19a680ba488d2aaa5bbd32519cbcc03b"
server_key_iv: "f5f1d714f615ab8bf472dd6230edf61f"
server_key_version: "encryptionkey@0"
I20260620 20:09:48.217271 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "2b9bf26fb26c4461a7dd8e9412c495b5"
format_stamp: "Formatted at 2026-06-20 20:09:48 on dist-test-slave-4bwq"
server_key: "19a680ba488d2aaa5bbd32519cbcc03b"
server_key_iv: "f5f1d714f615ab8bf472dd6230edf61f"
server_key_version: "encryptionkey@0"
I20260620 20:09:48.222287 8354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:09:48.225656 8369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:48.226940 8354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:09:48.227140 8354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "2b9bf26fb26c4461a7dd8e9412c495b5"
format_stamp: "Formatted at 2026-06-20 20:09:48 on dist-test-slave-4bwq"
server_key: "19a680ba488d2aaa5bbd32519cbcc03b"
server_key_iv: "f5f1d714f615ab8bf472dd6230edf61f"
server_key_version: "encryptionkey@0"
I20260620 20:09:48.227340 8354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:48.255610 8354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:48.259491 8354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:48.259786 8354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:48.277909 8354 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:34767
I20260620 20:09:48.277940 8420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:34767 every 8 connection(s)
I20260620 20:09:48.279450 8354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:48.281947 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 8354
I20260620 20:09:48.282169 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:48.282850 4774 external_mini_cluster.cc:1428] Setting key 338caa9062a700807197187bb696ea11
I20260620 20:09:48.283512 8421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:48.297634 8421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5: Bootstrap starting.
Jun 20 20:09:48 dist-test-slave-4bwq krb5kdc[8341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:48.300783 8421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:48.302110 8421 log.cc:824] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:48.305378 8421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5: No bootstrap required, opened a new log
I20260620 20:09:48.307257 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:48.284872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.4.169.190:34767)
0620 20:09:48.285678 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:48.285691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:48.285729 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:09:48.288023 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:48.288045 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:48.288482 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:48.288816 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:48.290611 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:48.291857 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:48.293617 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:48.294339 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:48.294564 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:48.299223 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:48.299230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:48.299246 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:48.299285 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:48.302911 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:48.303726 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:48.303734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:48.303741 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:48.303958 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:48.304305 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:48.304310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:48.304313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:48.304975 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:48.305266 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:48.305529 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 20:09:48.305969 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":387,"threads_started":1}
I20260620 20:09:48.313095 8421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9bf26fb26c4461a7dd8e9412c495b5" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34767 } }
I20260620 20:09:48.313405 8421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:48.313478 8421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9bf26fb26c4461a7dd8e9412c495b5, State: Initialized, Role: FOLLOWER
I20260620 20:09:48.313930 8421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9bf26fb26c4461a7dd8e9412c495b5" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34767 } }
I20260620 20:09:48.314105 8421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:48.314190 8421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:48.314309 8421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:48.316336 8421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9bf26fb26c4461a7dd8e9412c495b5" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34767 } }
I20260620 20:09:48.316747 8421 leader_election.cc:302] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b9bf26fb26c4461a7dd8e9412c495b5; no voters:
I20260620 20:09:48.317267 8421 leader_election.cc:288] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:48.317440 8426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:48.318105 8426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9bf26fb26c4461a7dd8e9412c495b5, State: Running, Role: LEADER
I20260620 20:09:48.318652 8426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9bf26fb26c4461a7dd8e9412c495b5" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34767 } }
I20260620 20:09:48.319343 8421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:48.321137 8427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b9bf26fb26c4461a7dd8e9412c495b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9bf26fb26c4461a7dd8e9412c495b5" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34767 } } }
I20260620 20:09:48.321524 8427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:48.323113 8431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:48.323800 8428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b9bf26fb26c4461a7dd8e9412c495b5. Latest consensus state: current_term: 1 leader_uuid: "2b9bf26fb26c4461a7dd8e9412c495b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9bf26fb26c4461a7dd8e9412c495b5" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34767 } } }
I20260620 20:09:48.324143 8428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:48.328394 8431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:48.336987 8431 catalog_manager.cc:1347] Generated new cluster ID: e25f8a6f80154d1981b39d9000aa288a
I20260620 20:09:48.337090 8431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:48.380897 8431 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:48.382710 8431 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:48.389267 8431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b9bf26fb26c4461a7dd8e9412c495b5: Generated new TSK 0
I20260620 20:09:48.390177 8431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:09:48.453528 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34767
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:48.785521 8447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:48.785830 8447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:48.785959 8447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:48.793651 8447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:48.793844 8447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:48.794072 8447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:09:48.803838 8447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:48.805217 8447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:48.806823 8447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:48.819947 8453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:48.819983 8455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:48.819950 8452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:48.820989 8447 server_base.cc:1034] running on GCE node
I20260620 20:09:48.821694 8447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:48.822695 8447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:48.823911 8447 hybrid_clock.cc:648] HybridClock initialized: now 1781986188823875 us; error 54 us; skew 500 ppm
I20260620 20:09:48.824290 8447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:48 dist-test-slave-4bwq krb5kdc[8341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986188, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:48.827405 8447 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:09:48.828706 8447 webserver.cc:460] Webserver started at http://127.4.169.129:37365/ using document root <none> and password file <none>
I20260620 20:09:48.829330 8447 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:48.829428 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:48.829677 8447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:48.832525 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "a91beb70753348a4a6cd580a1d5b78e8"
format_stamp: "Formatted at 2026-06-20 20:09:48 on dist-test-slave-4bwq"
server_key: "13ac60e208cc904e31cf13e0dca0aa56"
server_key_iv: "48d5450727655578ab5e017c8ca9ad1f"
server_key_version: "encryptionkey@0"
I20260620 20:09:48.833252 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "a91beb70753348a4a6cd580a1d5b78e8"
format_stamp: "Formatted at 2026-06-20 20:09:48 on dist-test-slave-4bwq"
server_key: "13ac60e208cc904e31cf13e0dca0aa56"
server_key_iv: "48d5450727655578ab5e017c8ca9ad1f"
server_key_version: "encryptionkey@0"
I20260620 20:09:48.838232 8447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:09:48.841645 8462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:48.842991 8447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:09:48.843156 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "a91beb70753348a4a6cd580a1d5b78e8"
format_stamp: "Formatted at 2026-06-20 20:09:48 on dist-test-slave-4bwq"
server_key: "13ac60e208cc904e31cf13e0dca0aa56"
server_key_iv: "48d5450727655578ab5e017c8ca9ad1f"
server_key_version: "encryptionkey@0"
I20260620 20:09:48.843353 8447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:48.855365 8447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:48.858956 8447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:48.859233 8447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:48.860090 8447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:48.861303 8447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:48.861374 8447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:48.861539 8447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:48.861598 8447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:48.893373 8447 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:38295
I20260620 20:09:48.893416 8574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:38295 every 8 connection(s)
I20260620 20:09:48.894678 8447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:09:48.904381 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 8447
I20260620 20:09:48.904583 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:09:48.905366 4774 external_mini_cluster.cc:1428] Setting key 39864ac822e6ba641be539caf68a807c
Jun 20 20:09:48 dist-test-slave-4bwq krb5kdc[8341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986188, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:48.915891 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:48.897027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:47101 (local address 127.4.169.190:34767)
0620 20:09:48.897453 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:48.897459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:48.898443 (+ 984us) server_negotiation.cc:407] Connection header received
0620 20:09:48.900262 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:48.900266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:48.900334 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:48.900427 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:48.901801 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:48.902569 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:48.904151 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:48.904821 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:48.904953 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:48.909145 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:48.909155 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:48.909163 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:48.909211 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:48.912760 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:48.913516 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:48.913524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:48.913529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:48.913657 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:48.914167 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:48.914169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:48.914171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:48.914489 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:48.914593 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:48.915485 (+ 892us) server_negotiation.cc:299] Negotiation successful
0620 20:09:48.915636 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260620 20:09:48.917518 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:48.897581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34767 (local address 127.4.169.129:47101)
0620 20:09:48.898258 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:48.898295 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:48.900022 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:48.900635 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:48.900643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:48.901111 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:48.901664 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:48.901680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:48.902751 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:48.902756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:48.904001 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:48.904013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:48.905058 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:48.905063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:48.905282 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:48.906029 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:48.906062 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:48.908913 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:48.912903 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:48.912909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:48.912922 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:48.913393 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:48.913769 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:48.913777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:48.913782 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:48.914046 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:48.914612 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:48.914623 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:48.915090 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 20:09:48.915413 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 20:09:48.915998 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":261,"threads_started":1}
I20260620 20:09:48.919351 8575 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34767
I20260620 20:09:48.919705 8575 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:48.920495 8575 heartbeater.cc:511] Master 127.4.169.190:34767 requested a full tablet report, sending...
I20260620 20:09:48.922775 8386 ts_manager.cc:194] Registered new tserver with Master: a91beb70753348a4a6cd580a1d5b78e8 (127.4.169.129:38295)
I20260620 20:09:48.924335 8386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:47101
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:09:48.949818 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34767
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:49.265136 8581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:49.265444 8581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:49.265618 8581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:49.272603 8581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:49.272753 8581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:49.272945 8581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:09:49.286142 8581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:49.287763 8581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:49.289292 8581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:49.303593 8587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:49.303610 8589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:49.303609 8586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:49.304327 8581 server_base.cc:1034] running on GCE node
I20260620 20:09:49.304934 8581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:49.305825 8581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:49.307039 8581 hybrid_clock.cc:648] HybridClock initialized: now 1781986189306996 us; error 54 us; skew 500 ppm
I20260620 20:09:49.307358 8581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:49 dist-test-slave-4bwq krb5kdc[8341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986189, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:49.310540 8581 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:09:49.311887 8581 webserver.cc:460] Webserver started at http://127.4.169.130:37129/ using document root <none> and password file <none>
I20260620 20:09:49.312534 8581 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:49.312677 8581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:49.312979 8581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:49.315778 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "538d774905d34b978f41315207466032"
format_stamp: "Formatted at 2026-06-20 20:09:49 on dist-test-slave-4bwq"
server_key: "f378e0ec150037970c177335cc4c4f86"
server_key_iv: "1ee0d27640e99ae676f79d004cc31c8b"
server_key_version: "encryptionkey@0"
I20260620 20:09:49.316526 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "538d774905d34b978f41315207466032"
format_stamp: "Formatted at 2026-06-20 20:09:49 on dist-test-slave-4bwq"
server_key: "f378e0ec150037970c177335cc4c4f86"
server_key_iv: "1ee0d27640e99ae676f79d004cc31c8b"
server_key_version: "encryptionkey@0"
I20260620 20:09:49.321486 8581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:09:49.325057 8596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:49.326478 8581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:09:49.326644 8581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "538d774905d34b978f41315207466032"
format_stamp: "Formatted at 2026-06-20 20:09:49 on dist-test-slave-4bwq"
server_key: "f378e0ec150037970c177335cc4c4f86"
server_key_iv: "1ee0d27640e99ae676f79d004cc31c8b"
server_key_version: "encryptionkey@0"
I20260620 20:09:49.326849 8581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:49.345352 8581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:49.349442 8581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:49.349727 8581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:49.350698 8581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:49.351944 8581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:49.352051 8581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:49.352202 8581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:49.352275 8581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:49.384227 8581 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:40497
I20260620 20:09:49.384310 8708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:40497 every 8 connection(s)
I20260620 20:09:49.385457 8581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:09:49.386809 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 8581
I20260620 20:09:49.386983 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:09:49.387674 4774 external_mini_cluster.cc:1428] Setting key d952cac63f2a1dbd263d591fe66665ac
Jun 20 20:09:49 dist-test-slave-4bwq krb5kdc[8341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986189, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:09:49.407281 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:49.388216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:36701 (local address 127.4.169.190:34767)
0620 20:09:49.388386 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:49.388390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:49.389918 (+ 1528us) server_negotiation.cc:407] Connection header received
0620 20:09:49.391769 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:49.391773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:49.391854 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:49.391974 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:49.393436 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.394136 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.395706 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.396445 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.396585 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:49.399947 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:49.399956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:49.399964 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:49.400007 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:49.403853 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:49.404646 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:49.404654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:49.404660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:49.404821 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:49.405269 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:49.405275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:49.405278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:49.405691 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:49.405853 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:49.406602 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 20:09:49.406851 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:09:49.408205 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:49.388839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34767 (local address 127.4.169.130:36701)
0620 20:09:49.389734 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:49.389790 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:49.391501 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:49.392206 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:49.392214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:49.392715 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:49.393278 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:49.393295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.394302 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.394306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:49.395562 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:49.395572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.396672 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.396677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:49.396882 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:49.397567 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:49.397594 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:49.399728 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:49.404028 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:49.404036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:49.404050 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:49.404497 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:49.404943 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:49.404954 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:49.404957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:49.405137 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:49.405827 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:49.405833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:49.406176 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 20:09:49.406475 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 20:09:49.406910 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":368,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
I20260620 20:09:49.409929 8709 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34767
I20260620 20:09:49.410248 8709 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:49.411057 8709 heartbeater.cc:511] Master 127.4.169.190:34767 requested a full tablet report, sending...
I20260620 20:09:49.412570 8386 ts_manager.cc:194] Registered new tserver with Master: 538d774905d34b978f41315207466032 (127.4.169.130:40497)
I20260620 20:09:49.413395 8386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:36701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:09:49.433413 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34767
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:49.756083 8714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:49.756398 8714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:49.756510 8714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:49.763521 8714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:49.763630 8714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:49.763742 8714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:09:49.773303 8714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:49.774564 8714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:49.775800 8714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:49.788968 8719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:49.789222 8720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:49.789258 8722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:49.789951 8714 server_base.cc:1034] running on GCE node
I20260620 20:09:49.790578 8714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:49.791474 8714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:49.792680 8714 hybrid_clock.cc:648] HybridClock initialized: now 1781986189792634 us; error 35 us; skew 500 ppm
I20260620 20:09:49.792946 8714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:49 dist-test-slave-4bwq krb5kdc[8341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986189, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:49.795943 8714 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:09:49.797494 8714 webserver.cc:460] Webserver started at http://127.4.169.131:33551/ using document root <none> and password file <none>
I20260620 20:09:49.798243 8714 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:49.798452 8714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:49.798779 8714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:49.801759 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "7486c399a4d241cb8ca9b85e0bf7795e"
format_stamp: "Formatted at 2026-06-20 20:09:49 on dist-test-slave-4bwq"
server_key: "cb848c48b635e3b3c3ec6adb535da542"
server_key_iv: "d9a8f00cd5b3c4a34149902188879760"
server_key_version: "encryptionkey@0"
I20260620 20:09:49.802620 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "7486c399a4d241cb8ca9b85e0bf7795e"
format_stamp: "Formatted at 2026-06-20 20:09:49 on dist-test-slave-4bwq"
server_key: "cb848c48b635e3b3c3ec6adb535da542"
server_key_iv: "d9a8f00cd5b3c4a34149902188879760"
server_key_version: "encryptionkey@0"
I20260620 20:09:49.807672 8714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:09:49.811168 8729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:49.812577 8714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:49.812712 8714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "7486c399a4d241cb8ca9b85e0bf7795e"
format_stamp: "Formatted at 2026-06-20 20:09:49 on dist-test-slave-4bwq"
server_key: "cb848c48b635e3b3c3ec6adb535da542"
server_key_iv: "d9a8f00cd5b3c4a34149902188879760"
server_key_version: "encryptionkey@0"
I20260620 20:09:49.812877 8714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:49.855087 8714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:49.859215 8714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:49.859455 8714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:49.860314 8714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:49.861660 8714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:49.861728 8714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:49.861807 8714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:49.861850 8714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:49.894688 8714 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:40189
I20260620 20:09:49.894690 8841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:40189 every 8 connection(s)
I20260620 20:09:49.896214 8714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:09:49.901520 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 8714
I20260620 20:09:49.901691 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:09:49.902093 4774 external_mini_cluster.cc:1428] Setting key e1aea6629c1fc999e9c640f179778f68
Jun 20 20:09:49 dist-test-slave-4bwq krb5kdc[8341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986189, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:49.915599 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:49.898938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:49915 (local address 127.4.169.190:34767)
0620 20:09:49.899113 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:49.899117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:49.900346 (+ 1229us) server_negotiation.cc:407] Connection header received
0620 20:09:49.902219 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:49.902226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:49.902344 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:49.902492 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:49.904166 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.905018 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.906521 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.906900 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.906994 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:49.910196 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:49.910202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:49.910205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:49.910235 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:49.912678 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:49.913357 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:49.913362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:49.913366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:49.913488 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:49.913889 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:49.913896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:49.913900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:49.914160 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:49.914312 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:49.915037 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 20:09:49.915244 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:09:49.916730 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:49.899459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34767 (local address 127.4.169.131:49915)
0620 20:09:49.900177 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:49.900218 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:49.901955 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:49.902690 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:49.902701 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:49.903259 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:49.903987 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:49.904008 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.905165 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.905169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:49.906346 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:49.906362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.907073 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.907078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:49.907288 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:49.907892 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:49.907913 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:49.910000 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:49.912808 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:49.912816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:49.912832 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:49.913232 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:49.913593 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:49.913599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:49.913603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:49.913782 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:49.914294 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:49.914302 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:49.914678 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 20:09:49.914937 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 20:09:49.915423 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
I20260620 20:09:49.918049 8842 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34767
I20260620 20:09:49.918298 8842 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:49.918980 8842 heartbeater.cc:511] Master 127.4.169.190:34767 requested a full tablet report, sending...
I20260620 20:09:49.920318 8386 ts_manager.cc:194] Registered new tserver with Master: 7486c399a4d241cb8ca9b85e0bf7795e (127.4.169.131:40189)
I20260620 20:09:49.921180 8386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:49915
I20260620 20:09:49.927541 8575 heartbeater.cc:503] Master 127.4.169.190:34767 was elected leader, sending a full tablet report...
I20260620 20:09:49.929497 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:09:49 dist-test-slave-4bwq krb5kdc[8341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:09:49 dist-test-slave-4bwq krb5kdc[8341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:49.976392 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:49.961331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.4.169.129:38295)
0620 20:09:49.962012 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:49.962019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:49.962044 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:09:49.962102 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:49.962108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:49.962334 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:49.962564 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:49.964009 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.965019 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.966260 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.966928 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.967062 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:49.970907 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:49.970917 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:49.970931 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:49.970966 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:49.973625 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:49.974281 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:49.974287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:49.974289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:49.974464 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:49.974825 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:49.974831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:49.974834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:49.975156 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:49.975450 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:49.975767 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 20:09:49.976028 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":321,"threads_started":1}
Jun 20 20:09:49 dist-test-slave-4bwq krb5kdc[8341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:49.993086 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:49.980549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.4.169.190:34767)
0620 20:09:49.980797 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:49.980802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:49.980884 (+ 82us) server_negotiation.cc:407] Connection header received
0620 20:09:49.981039 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:49.981043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:49.981106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:49.981197 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:49.982342 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.983360 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.984531 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.984922 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.985027 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:49.988560 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:49.988565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:49.988568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:49.988599 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:49.990921 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:49.991501 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:49.991504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:49.991507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:49.991615 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:49.991942 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:49.991945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:49.991946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:49.992278 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:49.992447 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:49.992722 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 20:09:49.992868 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260620 20:09:49.993638 8386 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57766
I20260620 20:09:49.994670 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8447
I20260620 20:09:50.006850 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:49.995577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.4.169.129:38295)
0620 20:09:49.995789 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:49.995797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:49.995918 (+ 121us) server_negotiation.cc:407] Connection header received
0620 20:09:49.996041 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:49.996044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:49.996100 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:49.996176 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:49.997281 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.997943 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.999151 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:49.999549 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:49.999646 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:50.001797 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:50.001802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:50.001805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:50.001836 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:50.004499 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:50.005162 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:50.005166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:50.005168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:50.005291 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:50.005580 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:50.005583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:50.005584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:50.005813 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:50.005976 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:50.006251 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 20:09:50.006499 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:09:50.007560 8570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:50.088971 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 8447
I20260620 20:09:50.104701 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8581
Jun 20 20:09:50 dist-test-slave-4bwq krb5kdc[8341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:09:50.120780 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:50.105563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.4.169.130:40497)
0620 20:09:50.106188 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:50.106196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:50.106223 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:09:50.106291 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:50.106299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:50.106581 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:50.106723 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:50.107920 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:50.108938 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:50.110126 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:50.110736 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:50.110848 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:50.114930 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:50.114941 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:50.114954 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:50.114989 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:50.117919 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:50.118636 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:50.118643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:50.118646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:50.118775 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:50.119232 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:50.119236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:50.119239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:50.119552 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:50.119859 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:50.120184 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 20:09:50.120450 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":345,"threads_started":1}
I20260620 20:09:50.122030 8704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:50.204521 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 8581
I20260620 20:09:50.220072 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8714
Jun 20 20:09:50 dist-test-slave-4bwq krb5kdc[8341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:09:50.235651 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:50.221081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.4.169.131:40189)
0620 20:09:50.221745 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:50.221752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:50.221781 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:09:50.221837 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:50.221843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:50.222046 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:50.222167 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:50.223377 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:50.224383 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:50.225753 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:50.226281 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:50.226377 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:50.230580 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:50.230591 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:50.230605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:50.230635 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:50.233289 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:50.233881 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:50.233887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:50.233890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:50.234002 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:50.234324 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:50.234326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:50.234328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:50.234739 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:50.234966 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:50.235200 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 20:09:50.235390 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":388,"threads_started":1}
I20260620 20:09:50.236888 8837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:50.317900 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 8714
I20260620 20:09:50.333483 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8354
I20260620 20:09:50.334592 8416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:50.435647 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 8354
2026-06-20T20:09:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3897 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:09:50 dist-test-slave-4bwq krb5kdc[8861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:09:50 dist-test-slave-4bwq krb5kdc[8861](info): set up 2 sockets
Jun 20 20:09:50 dist-test-slave-4bwq krb5kdc[8861](info): commencing operation
krb5kdc: starting...
W20260620 20:09:51.523077 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:09:51 dist-test-slave-4bwq krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:09:51.711306 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:40711
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:40711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:52.031203 8874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:52.031467 8874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:52.031558 8874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:52.038439 8874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:52.038532 8874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:52.038576 8874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:52.038638 8874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:52.038687 8874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:52.048102 8874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:40711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:40711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:52.049232 8874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:52.050453 8874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:52.061547 8879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:52.061584 8882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:52.061621 8880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:52.062237 8874 server_base.cc:1034] running on GCE node
I20260620 20:09:52.062997 8874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:52.065460 8874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:52.066670 8874 hybrid_clock.cc:648] HybridClock initialized: now 1781986192066611 us; error 51 us; skew 500 ppm
I20260620 20:09:52.067042 8874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:52 dist-test-slave-4bwq krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:52.070222 8874 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:09:52.071424 8874 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:52.071511 8874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:52.071755 8874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:52.074649 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "ba4ceb83a88948a99e0465c21ef09b6f"
format_stamp: "Formatted at 2026-06-20 20:09:52 on dist-test-slave-4bwq"
server_key: "2ec6966c9048b6908fe54dd39a6fb3ef"
server_key_iv: "e8c534e98827b5b39233320d0c07d604"
server_key_version: "encryptionkey@0"
I20260620 20:09:52.075397 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "ba4ceb83a88948a99e0465c21ef09b6f"
format_stamp: "Formatted at 2026-06-20 20:09:52 on dist-test-slave-4bwq"
server_key: "2ec6966c9048b6908fe54dd39a6fb3ef"
server_key_iv: "e8c534e98827b5b39233320d0c07d604"
server_key_version: "encryptionkey@0"
I20260620 20:09:52.080361 8874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:09:52.083627 8888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:52.084970 8874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:52.085168 8874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "ba4ceb83a88948a99e0465c21ef09b6f"
format_stamp: "Formatted at 2026-06-20 20:09:52 on dist-test-slave-4bwq"
server_key: "2ec6966c9048b6908fe54dd39a6fb3ef"
server_key_iv: "e8c534e98827b5b39233320d0c07d604"
server_key_version: "encryptionkey@0"
I20260620 20:09:52.085399 8874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:52.114331 8874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:52.118793 8874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:52.119071 8874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:52.136735 8874 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:40711
I20260620 20:09:52.136756 8939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:40711 every 8 connection(s)
I20260620 20:09:52.137603 8874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:52.138799 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 8874
I20260620 20:09:52.139005 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:52.139787 4774 external_mini_cluster.cc:1428] Setting key 04ecbc46ba629cbaa5cf67f9b04599c5
I20260620 20:09:52.142112 8940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 20:09:52 dist-test-slave-4bwq krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:52.154966 8940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f: Bootstrap starting.
I20260620 20:09:52.158002 8940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:52.159052 8940 log.cc:824] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:52.161742 8940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f: No bootstrap required, opened a new log
I20260620 20:09:52.162300 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:52.141787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.4.169.190:40711)
0620 20:09:52.142584 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:52.142595 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:52.142633 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:09:52.144170 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:52.144191 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:52.144557 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:52.144871 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:52.146738 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:52.148058 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:52.149796 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:52.150514 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:52.150638 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:52.155113 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:52.155125 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:52.155144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:52.155190 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:52.158176 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:52.158856 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:52.158864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:52.158879 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:52.159045 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:52.159427 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:52.159438 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:52.159440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:52.160225 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:52.160515 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:52.160949 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 20:09:52.161319 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":387,"threads_started":1}
I20260620 20:09:52.169080 8940 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4ceb83a88948a99e0465c21ef09b6f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40711 } }
I20260620 20:09:52.169507 8940 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:52.169673 8940 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4ceb83a88948a99e0465c21ef09b6f, State: Initialized, Role: FOLLOWER
I20260620 20:09:52.170260 8940 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4ceb83a88948a99e0465c21ef09b6f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40711 } }
I20260620 20:09:52.170557 8940 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:52.170699 8940 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:52.170879 8940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:52.172881 8940 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4ceb83a88948a99e0465c21ef09b6f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40711 } }
I20260620 20:09:52.173341 8940 leader_election.cc:302] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba4ceb83a88948a99e0465c21ef09b6f; no voters:
I20260620 20:09:52.173871 8940 leader_election.cc:288] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:52.173995 8945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:52.174670 8945 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [term 1 LEADER]: Becoming Leader. State: Replica: ba4ceb83a88948a99e0465c21ef09b6f, State: Running, Role: LEADER
I20260620 20:09:52.175158 8945 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4ceb83a88948a99e0465c21ef09b6f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40711 } }
I20260620 20:09:52.175858 8940 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:52.177801 8946 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba4ceb83a88948a99e0465c21ef09b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4ceb83a88948a99e0465c21ef09b6f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40711 } } }
I20260620 20:09:52.177879 8947 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4ceb83a88948a99e0465c21ef09b6f. Latest consensus state: current_term: 1 leader_uuid: "ba4ceb83a88948a99e0465c21ef09b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4ceb83a88948a99e0465c21ef09b6f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40711 } } }
I20260620 20:09:52.178252 8947 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:52.178258 8946 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:52.179251 8952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:52.185408 8952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:52.193276 8952 catalog_manager.cc:1347] Generated new cluster ID: 963933a13fdf4b7daa1669b67f7b09e5
I20260620 20:09:52.193378 8952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:52.204074 8952 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:52.205338 8952 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:52.215651 8952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f: Generated new TSK 0
I20260620 20:09:52.216457 8952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:09:52.267724 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40711
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:52.588145 8966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:52.588420 8966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:52.588518 8966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:52.596084 8966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:52.596196 8966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:52.596308 8966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:09:52.605892 8966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:52.607136 8966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:52.608727 8966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:52.621654 8972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:52.621899 8974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:52.621662 8971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:52.622731 8966 server_base.cc:1034] running on GCE node
I20260620 20:09:52.623253 8966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:52.624094 8966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:52.625335 8966 hybrid_clock.cc:648] HybridClock initialized: now 1781986192625292 us; error 33 us; skew 500 ppm
I20260620 20:09:52.625626 8966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:52 dist-test-slave-4bwq krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:52.628878 8966 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:09:52.630125 8966 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:52.630280 8966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:52.630666 8966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:52.633667 8966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "c22d2407433c4cc6a396a7c8b97984de"
format_stamp: "Formatted at 2026-06-20 20:09:52 on dist-test-slave-4bwq"
server_key: "cd752c2ebf35081b18f1cb4965233279"
server_key_iv: "51967da22994b0617c375ad543d758ea"
server_key_version: "encryptionkey@0"
I20260620 20:09:52.634588 8966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "c22d2407433c4cc6a396a7c8b97984de"
format_stamp: "Formatted at 2026-06-20 20:09:52 on dist-test-slave-4bwq"
server_key: "cd752c2ebf35081b18f1cb4965233279"
server_key_iv: "51967da22994b0617c375ad543d758ea"
server_key_version: "encryptionkey@0"
I20260620 20:09:52.639668 8966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:09:52.643116 8980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:52.644431 8966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:52.644606 8966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "c22d2407433c4cc6a396a7c8b97984de"
format_stamp: "Formatted at 2026-06-20 20:09:52 on dist-test-slave-4bwq"
server_key: "cd752c2ebf35081b18f1cb4965233279"
server_key_iv: "51967da22994b0617c375ad543d758ea"
server_key_version: "encryptionkey@0"
I20260620 20:09:52.644821 8966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:52.672816 8966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:52.677488 8966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:52.677789 8966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:52.678910 8966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:52.680684 8966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:52.680897 8966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:52.681079 8966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:52.681186 8966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:52.714154 8966 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:36725
I20260620 20:09:52.714205 9092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:36725 every 8 connection(s)
I20260620 20:09:52.715093 8966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:09:52.725582 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 8966
I20260620 20:09:52.725812 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:09:52.726589 4774 external_mini_cluster.cc:1428] Setting key e75f0604951f223132dbe1634f091853
Jun 20 20:09:52 dist-test-slave-4bwq krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:52.735532 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:52.717584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:49445 (local address 127.4.169.190:40711)
0620 20:09:52.717995 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:52.717999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:52.719019 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 20:09:52.720830 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:52.720834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:52.720922 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:52.721038 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:52.722586 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:52.723297 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:52.724846 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:52.725383 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:52.725529 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:52.729603 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:52.729608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:52.729611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:52.729639 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:52.732090 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:52.733016 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:52.733020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:52.733023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:52.733130 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:52.733593 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:52.733596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:52.733598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:52.733824 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:52.733926 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:52.734822 (+ 896us) server_negotiation.cc:299] Negotiation successful
0620 20:09:52.735066 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":186,"threads_started":1}
I20260620 20:09:52.736162 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:52.718163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40711 (local address 127.4.169.129:49445)
0620 20:09:52.718870 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:52.718908 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:52.720546 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:52.721277 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:52.721285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:52.721819 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:52.722370 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:52.722431 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:52.723451 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:52.723456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:52.724682 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:52.724695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:52.725622 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:52.725627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:52.725948 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:52.726684 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:52.726725 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:52.729398 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:52.732237 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:52.732249 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:52.732268 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:52.732899 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:52.733237 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:52.733244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:52.733251 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:52.733485 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:52.733947 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:52.733957 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:52.734374 (+ 417us) client_negotiation.cc:769] Sending connection context
0620 20:09:52.734674 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 20:09:52.735049 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":306,"threads_started":1}
I20260620 20:09:52.737812 9093 heartbeater.cc:347] Connected to a master server at 127.4.169.190:40711
I20260620 20:09:52.738107 9093 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:52.738792 9093 heartbeater.cc:511] Master 127.4.169.190:40711 requested a full tablet report, sending...
I20260620 20:09:52.740974 8905 ts_manager.cc:194] Registered new tserver with Master: c22d2407433c4cc6a396a7c8b97984de (127.4.169.129:36725)
I20260620 20:09:52.743193 8905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:49445
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:09:52.771028 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40711
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:53.091900 9099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:53.092234 9099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:53.092406 9099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:53.099597 9099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:53.099761 9099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:53.099939 9099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:09:53.110049 9099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:53.111374 9099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:53.112648 9099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:53.125640 9104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:53.125746 9105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:53.125720 9107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:53.126664 9099 server_base.cc:1034] running on GCE node
I20260620 20:09:53.127305 9099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:53.128305 9099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:53.129500 9099 hybrid_clock.cc:648] HybridClock initialized: now 1781986193129458 us; error 41 us; skew 500 ppm
I20260620 20:09:53.129828 9099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:53 dist-test-slave-4bwq krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986193, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:53.133009 9099 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:09:53.134161 9099 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:53.134254 9099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:53.134588 9099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:53.137494 9099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "459bfef8dfe24c07861fb430352b0786"
format_stamp: "Formatted at 2026-06-20 20:09:53 on dist-test-slave-4bwq"
server_key: "55d74bcce1da50c128ab0bb4583b0082"
server_key_iv: "2bddd3fd167985c4b996cc40b61f31b5"
server_key_version: "encryptionkey@0"
I20260620 20:09:53.138303 9099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "459bfef8dfe24c07861fb430352b0786"
format_stamp: "Formatted at 2026-06-20 20:09:53 on dist-test-slave-4bwq"
server_key: "55d74bcce1da50c128ab0bb4583b0082"
server_key_iv: "2bddd3fd167985c4b996cc40b61f31b5"
server_key_version: "encryptionkey@0"
I20260620 20:09:53.143700 9099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:09:53.147384 9113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:53.148697 9099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:53.148926 9099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "459bfef8dfe24c07861fb430352b0786"
format_stamp: "Formatted at 2026-06-20 20:09:53 on dist-test-slave-4bwq"
server_key: "55d74bcce1da50c128ab0bb4583b0082"
server_key_iv: "2bddd3fd167985c4b996cc40b61f31b5"
server_key_version: "encryptionkey@0"
I20260620 20:09:53.149137 9099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:53.172538 9099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:53.176764 9099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:53.177049 9099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:53.178017 9099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:53.179320 9099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:53.179445 9099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:53.179580 9099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:53.179668 9099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:53.211505 9099 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:43719
I20260620 20:09:53.211586 9225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:43719 every 8 connection(s)
I20260620 20:09:53.212353 9099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:09:53.220688 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 9099
I20260620 20:09:53.220865 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:09:53.221410 4774 external_mini_cluster.cc:1428] Setting key 7ffd61e6cbf07aeb0281219e72112aa8
Jun 20 20:09:53 dist-test-slave-4bwq krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986193, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:53.237819 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.214766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:46881 (local address 127.4.169.190:40711)
0620 20:09:53.214930 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:53.214934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:53.216018 (+ 1084us) server_negotiation.cc:407] Connection header received
0620 20:09:53.217849 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:53.217855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:53.217940 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:53.218045 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:53.219511 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.220200 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.222421 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.222899 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.223031 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.231744 (+ 8713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:53.231749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:53.231752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:53.231793 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:53.234301 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:53.235354 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:53.235358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:53.235360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:53.235488 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:53.235999 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:53.236002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:53.236004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:53.236266 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:53.236377 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:53.237389 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0620 20:09:53.237565 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:09:53.239491 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.215202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40711 (local address 127.4.169.130:46881)
0620 20:09:53.215868 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:53.215906 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:53.217589 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:53.218256 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:53.218263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:53.218816 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:53.219344 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.219360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.220428 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.220435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.222222 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.222233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.223111 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.223118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.223406 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.228822 (+ 5416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:53.228844 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:53.231508 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:53.234497 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:53.234510 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:53.234531 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:53.235217 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:53.235603 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:53.235611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:53.235618 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:53.235877 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:53.236399 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:53.236410 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:53.236950 (+ 540us) client_negotiation.cc:769] Sending connection context
0620 20:09:53.237312 (+ 362us) client_negotiation.cc:240] Negotiation successful
0620 20:09:53.237952 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:09:53.241103 9226 heartbeater.cc:347] Connected to a master server at 127.4.169.190:40711
I20260620 20:09:53.241405 9226 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:53.242048 9226 heartbeater.cc:511] Master 127.4.169.190:40711 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
I20260620 20:09:53.243548 8905 ts_manager.cc:194] Registered new tserver with Master: 459bfef8dfe24c07861fb430352b0786 (127.4.169.130:43719)
I20260620 20:09:53.244329 8905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:46881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:09:53.265918 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40711
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:09:53.586424 9231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:53.586788 9231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:53.586968 9231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:53.594565 9231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:53.594658 9231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:53.594782 9231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:09:53.604138 9231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:53.605341 9231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:53.606705 9231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:53.619386 9237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:53.619467 9236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:53.619467 9239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:53.620452 9231 server_base.cc:1034] running on GCE node
I20260620 20:09:53.621011 9231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:53.621922 9231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:53.623153 9231 hybrid_clock.cc:648] HybridClock initialized: now 1781986193623104 us; error 53 us; skew 500 ppm
I20260620 20:09:53.623468 9231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:09:53 dist-test-slave-4bwq krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986193, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:09:53.626569 9231 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:09:53.627638 9231 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:53.627732 9231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:53.627990 9231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:53.630894 9231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "abe5f0394b8249ea9f87d92e4e34e87e"
format_stamp: "Formatted at 2026-06-20 20:09:53 on dist-test-slave-4bwq"
server_key: "ffcf8a12789e91d54020c2c43a824227"
server_key_iv: "a5cf27e238e8dc246d7a2c9b2bbb730a"
server_key_version: "encryptionkey@0"
I20260620 20:09:53.631634 9231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "abe5f0394b8249ea9f87d92e4e34e87e"
format_stamp: "Formatted at 2026-06-20 20:09:53 on dist-test-slave-4bwq"
server_key: "ffcf8a12789e91d54020c2c43a824227"
server_key_iv: "a5cf27e238e8dc246d7a2c9b2bbb730a"
server_key_version: "encryptionkey@0"
I20260620 20:09:53.636518 9231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:09:53.639736 9245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:53.641273 9231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:09:53.641399 9231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "abe5f0394b8249ea9f87d92e4e34e87e"
format_stamp: "Formatted at 2026-06-20 20:09:53 on dist-test-slave-4bwq"
server_key: "ffcf8a12789e91d54020c2c43a824227"
server_key_iv: "a5cf27e238e8dc246d7a2c9b2bbb730a"
server_key_version: "encryptionkey@0"
I20260620 20:09:53.641600 9231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:53.661315 9231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:53.665485 9231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:53.665706 9231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:53.666613 9231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:53.667899 9231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:53.667985 9231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:53.668057 9231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:53.668112 9231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:53.701325 9231 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:44893
I20260620 20:09:53.701427 9357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:44893 every 8 connection(s)
I20260620 20:09:53.702216 9231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:09:53.703805 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 9231
I20260620 20:09:53.703965 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:09:53.704394 4774 external_mini_cluster.cc:1428] Setting key d5e5a03852b4bbff6a0ae8ee10a8680d
Jun 20 20:09:53 dist-test-slave-4bwq krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986193, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:09:53.722882 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.705007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:47949 (local address 127.4.169.190:40711)
0620 20:09:53.705230 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:53.705237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:53.706693 (+ 1456us) server_negotiation.cc:407] Connection header received
0620 20:09:53.708607 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:53.708610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:53.708700 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:53.708787 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:53.710181 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.710957 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.712515 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.712898 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.712984 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.716690 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:53.716698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:53.716703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:53.716746 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:53.720079 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:53.720869 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:53.720873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:53.720875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:53.721003 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:53.721362 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:53.721365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:53.721367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:53.721657 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:53.721781 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:53.722464 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 20:09:53.722628 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:09:53.723838 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.705565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40711 (local address 127.4.169.131:47949)
0620 20:09:53.706503 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:53.706566 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:53.708371 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:53.708943 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:53.708951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:53.709475 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:53.710035 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.710052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.711108 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.711113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.712400 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.712412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.713068 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.713073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.713252 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.713985 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:09:53.714011 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:53.716487 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:53.720239 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:53.720248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:53.720266 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:53.720737 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:53.721099 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:09:53.721102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:09:53.721105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:09:53.721260 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:09:53.721789 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:53.721794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:53.722078 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 20:09:53.722298 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 20:09:53.722733 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":441,"threads_started":1}
I20260620 20:09:53.725224 9358 heartbeater.cc:347] Connected to a master server at 127.4.169.190:40711
I20260620 20:09:53.725543 9358 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:53.726197 9358 heartbeater.cc:511] Master 127.4.169.190:40711 requested a full tablet report, sending...
I20260620 20:09:53.727669 8905 ts_manager.cc:194] Registered new tserver with Master: abe5f0394b8249ea9f87d92e4e34e87e (127.4.169.131:44893)
I20260620 20:09:53.728462 8905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:47949
I20260620 20:09:53.731849 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:09:53.746131 9093 heartbeater.cc:503] Master 127.4.169.190:40711 was elected leader, sending a full tablet report...
I20260620 20:09:53.748239 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.735975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40652 (local address 127.4.169.190:40711)
0620 20:09:53.736193 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:53.736203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:53.736617 (+ 414us) server_negotiation.cc:407] Connection header received
0620 20:09:53.736757 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:53.736761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:53.736817 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:53.736895 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:53.738275 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.739073 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.740261 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.740812 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.740907 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.743064 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:53.743068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:53.743071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:53.743100 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:53.745534 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:53.746146 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:53.746152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:53.746155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:53.746304 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:53.746711 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:53.746717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:53.746727 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:53.747215 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:53.747441 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:53.747784 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 20:09:53.747960 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:09:53.751889 8905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:09:53.758689 8905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:53.786458 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.770888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:43719 (local address 127.0.0.1:43712)
0620 20:09:53.778654 (+ 7766us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:53.778678 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:53.778820 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:53.779421 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:53.779427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:53.779451 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:53.779812 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.779824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.781894 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.781899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.783706 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.783717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.785451 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.785457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.785637 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.785687 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 20:09:53.785831 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 20:09:53.786003 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7632,"mutex_wait_us":46,"thread_start_us":6311,"threads_started":1}
I20260620 20:09:53.786556 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.771251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.4.169.130:43719)
0620 20:09:53.777725 (+ 6474us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:53.777734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:53.778859 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 20:09:53.778967 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:53.778974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:53.779201 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:53.779415 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:53.780174 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.781744 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.783860 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.785244 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.785389 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.785587 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:53.786013 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 20:09:53.786191 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6308,"thread_start_us":6409,"threads_started":1}
I20260620 20:09:53.786922 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.769412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:44893 (local address 127.0.0.1:41566)
0620 20:09:53.770494 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:53.770537 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:53.770684 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:53.778946 (+ 8262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:53.778954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:53.778992 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:53.779359 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.779374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.781585 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.781592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.784018 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.784034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.785753 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.785760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.785953 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.785984 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:09:53.786069 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:09:53.786197 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":24,"thread_start_us":695,"threads_started":1}
I20260620 20:09:53.787334 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.770264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:36725 (local address 127.0.0.1:36884)
0620 20:09:53.777446 (+ 7182us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:53.777468 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:53.777592 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:53.779960 (+ 2368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:53.779964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:53.779989 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:53.780345 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.780356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.782293 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.782300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.784435 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:53.784447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.786610 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.786619 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:53.786793 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.786825 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:09:53.786895 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:09:53.787012 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7051,"mutex_wait_us":1,"thread_start_us":8127,"threads_started":1}
I20260620 20:09:53.788192 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.771016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.4.169.129:36725)
0620 20:09:53.779410 (+ 8394us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:53.779419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:53.779443 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:09:53.779533 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:53.779542 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:53.779769 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:53.779964 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:53.780523 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.782148 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.784607 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.786340 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.787397 (+ 1057us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.787621 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:53.787688 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 20:09:53.787858 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8219,"thread_start_us":8042,"threads_started":1}
I20260620 20:09:53.788195 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:53.769710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.4.169.131:44893)
0620 20:09:53.778289 (+ 8579us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:53.778295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:53.778320 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:09:53.778442 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:53.778451 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:53.778703 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:53.779556 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:53.779794 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.781443 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.784194 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:53.785565 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:53.785685 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:53.785906 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:53.787734 (+ 1828us) server_negotiation.cc:299] Negotiation successful
0620 20:09:53.787878 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8426,"thread_start_us":8302,"threads_started":1}
I20260620 20:09:53.790576 9161 tablet_service.cc:1467] Processing CreateTablet for tablet cf9eeb92fce14b278df8f3d3afc60517 (DEFAULT_TABLE table=test-table [id=cc47f4497f5047438023b0c4aad00b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:53.791432 9028 tablet_service.cc:1467] Processing CreateTablet for tablet cf9eeb92fce14b278df8f3d3afc60517 (DEFAULT_TABLE table=test-table [id=cc47f4497f5047438023b0c4aad00b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:53.792004 9161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9eeb92fce14b278df8f3d3afc60517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:53.791538 9293 tablet_service.cc:1467] Processing CreateTablet for tablet cf9eeb92fce14b278df8f3d3afc60517 (DEFAULT_TABLE table=test-table [id=cc47f4497f5047438023b0c4aad00b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:53.792537 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9eeb92fce14b278df8f3d3afc60517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:53.792742 9293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9eeb92fce14b278df8f3d3afc60517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:53.803007 9376 tablet_bootstrap.cc:492] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: Bootstrap starting.
I20260620 20:09:53.806110 9376 tablet_bootstrap.cc:654] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:53.806133 9377 tablet_bootstrap.cc:492] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: Bootstrap starting.
I20260620 20:09:53.807960 9378 tablet_bootstrap.cc:492] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: Bootstrap starting.
I20260620 20:09:53.807955 9376 log.cc:824] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:53.811291 9377 tablet_bootstrap.cc:654] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:53.811285 9378 tablet_bootstrap.cc:654] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:53.811806 9376 tablet_bootstrap.cc:492] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: No bootstrap required, opened a new log
I20260620 20:09:53.812057 9376 ts_tablet_manager.cc:1397] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260620 20:09:53.812445 9377 log.cc:824] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:53.812445 9378 log.cc:824] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:53.815124 9377 tablet_bootstrap.cc:492] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: No bootstrap required, opened a new log
I20260620 20:09:53.815397 9377 ts_tablet_manager.cc:1397] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 20:09:53.815434 9378 tablet_bootstrap.cc:492] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: No bootstrap required, opened a new log
I20260620 20:09:53.815673 9378 ts_tablet_manager.cc:1397] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 20:09:53.822221 9377 raft_consensus.cc:348] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } }
I20260620 20:09:53.822628 9377 raft_consensus.cc:374] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:53.822758 9377 raft_consensus.cc:729] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459bfef8dfe24c07861fb430352b0786, State: Initialized, Role: FOLLOWER
I20260620 20:09:53.823081 9376 raft_consensus.cc:348] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } }
I20260620 20:09:53.823464 9376 raft_consensus.cc:374] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:53.823349 9377 consensus_queue.cc:260] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } }
I20260620 20:09:53.823557 9376 raft_consensus.cc:729] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c22d2407433c4cc6a396a7c8b97984de, State: Initialized, Role: FOLLOWER
I20260620 20:09:53.824406 9226 heartbeater.cc:503] Master 127.4.169.190:40711 was elected leader, sending a full tablet report...
I20260620 20:09:53.824241 9376 consensus_queue.cc:260] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } }
I20260620 20:09:53.825907 9376 ts_tablet_manager.cc:1428] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260620 20:09:53.825898 9378 raft_consensus.cc:348] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } }
I20260620 20:09:53.826351 9378 raft_consensus.cc:374] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:53.826514 9378 raft_consensus.cc:729] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe5f0394b8249ea9f87d92e4e34e87e, State: Initialized, Role: FOLLOWER
I20260620 20:09:53.826864 9377 ts_tablet_manager.cc:1428] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20260620 20:09:53.827203 9378 consensus_queue.cc:260] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } }
I20260620 20:09:53.828528 9358 heartbeater.cc:503] Master 127.4.169.190:40711 was elected leader, sending a full tablet report...
I20260620 20:09:53.829214 9378 ts_tablet_manager.cc:1428] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260620 20:09:53.953891 9359 tablet.cc:2369] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:09:53.964143 9227 tablet.cc:2369] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:09:53.967324 9094 tablet.cc:2369] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:54.036073 9382 raft_consensus.cc:482] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:54.036396 9382 raft_consensus.cc:504] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } }
I20260620 20:09:54.038033 9382 leader_election.cc:288] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abe5f0394b8249ea9f87d92e4e34e87e (127.4.169.131:44893), c22d2407433c4cc6a396a7c8b97984de (127.4.169.129:36725)
I20260620 20:09:54.047288 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:54.038529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:44145 (local address 127.4.169.131:44893)
0620 20:09:54.038743 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:54.038751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:54.039462 (+ 711us) server_negotiation.cc:407] Connection header received
0620 20:09:54.039672 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:54.039678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:54.039755 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:54.039859 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:54.040735 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.041854 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.044898 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.046159 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.046327 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:54.046426 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:54.046881 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 20:09:54.047013 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:09:54.047284 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:54.038619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:39529 (local address 127.4.169.129:36725)
0620 20:09:54.038808 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:54.038816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:54.039460 (+ 644us) server_negotiation.cc:407] Connection header received
0620 20:09:54.039672 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:54.039678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:54.039755 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:54.039860 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:09:54.040735 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.041854 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.044898 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.046159 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.046326 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:54.046426 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:54.046881 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 20:09:54.047013 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:09:54.047698 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:54.038477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:44893 (local address 127.4.169.130:44145)
0620 20:09:54.039348 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:54.039366 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:54.039536 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:54.039888 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:54.039893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:54.039939 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:54.040510 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:54.040528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.041986 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.041991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:54.044731 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:54.044746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.046459 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.046467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:54.046693 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:54.046737 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 20:09:54.046837 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:09:54.047090 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":216,"spinlock_wait_cycles":42624,"thread_start_us":433,"threads_started":1}
I20260620 20:09:54.047731 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:54.038367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:36725 (local address 127.4.169.130:39529)
0620 20:09:54.039311 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:54.039336 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:54.039492 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:54.039888 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:54.039893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:54.039937 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:09:54.040572 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:54.040583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.041989 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.041993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:54.044733 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:54.044746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.046462 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.046484 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:54.046689 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:54.046742 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 20:09:54.046838 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 20:09:54.047090 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":391,"spinlock_wait_cycles":15872,"thread_start_us":603,"threads_started":1}
I20260620 20:09:54.048938 9313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9eeb92fce14b278df8f3d3afc60517" candidate_uuid: "459bfef8dfe24c07861fb430352b0786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" is_pre_election: true
I20260620 20:09:54.048939 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9eeb92fce14b278df8f3d3afc60517" candidate_uuid: "459bfef8dfe24c07861fb430352b0786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22d2407433c4cc6a396a7c8b97984de" is_pre_election: true
I20260620 20:09:54.049513 9313 raft_consensus.cc:2454] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 459bfef8dfe24c07861fb430352b0786 in term 0.
I20260620 20:09:54.049518 9048 raft_consensus.cc:2454] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 459bfef8dfe24c07861fb430352b0786 in term 0.
I20260620 20:09:54.050249 9116 leader_election.cc:302] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 459bfef8dfe24c07861fb430352b0786, abe5f0394b8249ea9f87d92e4e34e87e, c22d2407433c4cc6a396a7c8b97984de; no voters:
I20260620 20:09:54.050730 9382 raft_consensus.cc:2781] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:54.050877 9382 raft_consensus.cc:482] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:54.050994 9382 raft_consensus.cc:3037] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:54.053095 9382 raft_consensus.cc:504] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } }
I20260620 20:09:54.053738 9382 leader_election.cc:288] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [CANDIDATE]: Term 1 election: Requested vote from peers abe5f0394b8249ea9f87d92e4e34e87e (127.4.169.131:44893), c22d2407433c4cc6a396a7c8b97984de (127.4.169.129:36725)
I20260620 20:09:54.054170 9313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9eeb92fce14b278df8f3d3afc60517" candidate_uuid: "459bfef8dfe24c07861fb430352b0786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe5f0394b8249ea9f87d92e4e34e87e"
I20260620 20:09:54.054207 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9eeb92fce14b278df8f3d3afc60517" candidate_uuid: "459bfef8dfe24c07861fb430352b0786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22d2407433c4cc6a396a7c8b97984de"
I20260620 20:09:54.054368 9313 raft_consensus.cc:3037] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:54.054436 9048 raft_consensus.cc:3037] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:54.056535 9313 raft_consensus.cc:2454] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 459bfef8dfe24c07861fb430352b0786 in term 1.
I20260620 20:09:54.056540 9048 raft_consensus.cc:2454] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 459bfef8dfe24c07861fb430352b0786 in term 1.
I20260620 20:09:54.056988 9116 leader_election.cc:302] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 459bfef8dfe24c07861fb430352b0786, abe5f0394b8249ea9f87d92e4e34e87e, c22d2407433c4cc6a396a7c8b97984de; no voters:
I20260620 20:09:54.057272 9382 raft_consensus.cc:2781] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:54.057729 9382 raft_consensus.cc:686] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 1 LEADER]: Becoming Leader. State: Replica: 459bfef8dfe24c07861fb430352b0786, State: Running, Role: LEADER
I20260620 20:09:54.058295 9382 consensus_queue.cc:237] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } }
I20260620 20:09:54.062666 8903 catalog_manager.cc:5495] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 459bfef8dfe24c07861fb430352b0786 (127.4.169.130). New cstate: current_term: 1 leader_uuid: "459bfef8dfe24c07861fb430352b0786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "459bfef8dfe24c07861fb430352b0786" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 } health_report { overall_health: UNKNOWN } } }
I20260620 20:09:54.081439 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:54.076140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.4.169.130:43719)
0620 20:09:54.076341 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:54.076346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:54.076363 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:09:54.076527 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:54.076531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:54.076603 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:54.076730 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:09:54.077056 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.077791 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.079198 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.079616 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.079742 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:54.080200 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:09:54.080863 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:09:54.081032 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:54.081076 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 20:09:54.081141 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:09:54.089148 9048 raft_consensus.cc:1261] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [term 1 FOLLOWER]: Refusing update from remote peer 459bfef8dfe24c07861fb430352b0786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:54.089197 9313 raft_consensus.cc:1261] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [term 1 FOLLOWER]: Refusing update from remote peer 459bfef8dfe24c07861fb430352b0786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:54.090129 9387 consensus_queue.cc:1035] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22d2407433c4cc6a396a7c8b97984de" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:54.090533 9382 consensus_queue.cc:1035] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe5f0394b8249ea9f87d92e4e34e87e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 44893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:54.100098 9390 mvcc.cc:205] Tried to move back new op lower bound from 7299015450979991552 to 7299015450866577408. Current Snapshot: MvccSnapshot[applied={T|T < 7299015450979991552}]
I20260620 20:09:54.102470 9392 mvcc.cc:205] Tried to move back new op lower bound from 7299015450979991552 to 7299015450866577408. Current Snapshot: MvccSnapshot[applied={T|T < 7299015450979991552}]
I20260620 20:09:54.114470 8903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:54.114724 8903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:54.118139 8903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba4ceb83a88948a99e0465c21ef09b6f: Sending DeleteTablet for 3 replicas of tablet cf9eeb92fce14b278df8f3d3afc60517
I20260620 20:09:54.119223 9293 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9eeb92fce14b278df8f3d3afc60517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:54 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:41566
I20260620 20:09:54.119308 9161 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9eeb92fce14b278df8f3d3afc60517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:54 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:43712
I20260620 20:09:54.119594 9028 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9eeb92fce14b278df8f3d3afc60517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:54 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:36884
I20260620 20:09:54.120172 9399 tablet_replica.cc:330] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: stopping tablet replica
I20260620 20:09:54.120728 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8966
I20260620 20:09:54.120672 9399 raft_consensus.cc:2229] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:54.120806 9400 tablet_replica.cc:330] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: stopping tablet replica
I20260620 20:09:54.121150 9400 raft_consensus.cc:2229] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:54.121250 9399 raft_consensus.cc:2258] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:54.121595 9400 raft_consensus.cc:2258] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:54.124959 9399 ts_tablet_manager.cc:1905] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:54.127890 9400 ts_tablet_manager.cc:1905] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:54.134460 9399 ts_tablet_manager.cc:1918] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:09:54.134701 9399 log.cc:1196] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/cf9eeb92fce14b278df8f3d3afc60517
I20260620 20:09:54.134859 9401 tablet_replica.cc:330] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: stopping tablet replica
I20260620 20:09:54.135255 9401 raft_consensus.cc:2229] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:09:54.135489 9399 ts_tablet_manager.cc:1939] T cf9eeb92fce14b278df8f3d3afc60517 P abe5f0394b8249ea9f87d92e4e34e87e: Deleting consensus metadata
I20260620 20:09:54.135866 9401 raft_consensus.cc:2258] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:54.137916 9401 ts_tablet_manager.cc:1905] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 20:09:54 dist-test-slave-4bwq krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:09:54.146359 9400 ts_tablet_manager.cc:1918] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:09:54.146674 9400 log.cc:1196] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/cf9eeb92fce14b278df8f3d3afc60517
I20260620 20:09:54.147334 9400 ts_tablet_manager.cc:1939] T cf9eeb92fce14b278df8f3d3afc60517 P c22d2407433c4cc6a396a7c8b97984de: Deleting consensus metadata
I20260620 20:09:54.147325 8892 catalog_manager.cc:4841] TS abe5f0394b8249ea9f87d92e4e34e87e (127.4.169.131:44893): tablet cf9eeb92fce14b278df8f3d3afc60517 (table test-table [id=cc47f4497f5047438023b0c4aad00b84]) successfully deleted
I20260620 20:09:54.148936 9401 ts_tablet_manager.cc:1918] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:09:54.150321 8891 catalog_manager.cc:4841] TS c22d2407433c4cc6a396a7c8b97984de (127.4.169.129:36725): tablet cf9eeb92fce14b278df8f3d3afc60517 (table test-table [id=cc47f4497f5047438023b0c4aad00b84]) successfully deleted
I20260620 20:09:54.150765 9401 log.cc:1196] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/cf9eeb92fce14b278df8f3d3afc60517
I20260620 20:09:54.151382 9401 ts_tablet_manager.cc:1939] T cf9eeb92fce14b278df8f3d3afc60517 P 459bfef8dfe24c07861fb430352b0786: Deleting consensus metadata
I20260620 20:09:54.153122 8892 catalog_manager.cc:4841] TS 459bfef8dfe24c07861fb430352b0786 (127.4.169.130:43719): tablet cf9eeb92fce14b278df8f3d3afc60517 (table test-table [id=cc47f4497f5047438023b0c4aad00b84]) successfully deleted
I20260620 20:09:54.154853 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:54.122334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.4.169.129:36725)
0620 20:09:54.122513 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:54.122519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:54.136216 (+ 13697us) server_negotiation.cc:407] Connection header received
0620 20:09:54.136619 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:54.136627 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:54.136728 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:54.136818 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:54.138408 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.139478 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.141215 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.141822 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.141908 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:54.147151 (+ 5243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:54.147168 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:54.147185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:54.147237 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:54.151141 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:54.152081 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:54.152091 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:54.152095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:54.152247 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:54.152723 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:54.152728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:54.152730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:54.153259 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:54.153608 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:54.154059 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 20:09:54.154272 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:09:54.155442 9088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:54.267187 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 8966
I20260620 20:09:54.284606 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 9099
Jun 20 20:09:54 dist-test-slave-4bwq krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:09:54.301021 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:54.285375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.4.169.130:43719)
0620 20:09:54.285602 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:54.285609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:54.285630 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:09:54.285735 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:54.285741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:54.285827 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:54.285942 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:54.287314 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.288531 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.289852 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.290569 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.290694 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:54.294451 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:54.294467 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:54.294488 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:54.294534 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:54.298328 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:54.299035 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:54.299050 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:54.299053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:54.299184 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:54.299537 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:54.299542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:54.299547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:54.299893 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:54.300232 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:54.300459 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 20:09:54.300689 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 20:09:54.301782 9221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:54.411911 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 9099
I20260620 20:09:54.429710 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 9231
Jun 20 20:09:54 dist-test-slave-4bwq krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:09:54.443848 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:54.430489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.4.169.131:44893)
0620 20:09:54.430670 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:54.430675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:54.430692 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:09:54.430837 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:54.430841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:54.430909 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:54.430990 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:54.432136 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.432828 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.434009 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:54.434363 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:54.434660 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:54.438738 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:54.438753 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:54.438776 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:09:54.438807 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:54.441493 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:54.442068 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:54.442074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:54.442077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:54.442203 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:09:54.442546 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:09:54.442550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:09:54.442552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:09:54.442884 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:54.443094 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:54.443337 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 20:09:54.443501 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:09:54.444506 9353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:54.530874 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 9231
I20260620 20:09:54.546516 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 8874
I20260620 20:09:54.547598 8935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:54.655522 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 8874
2026-06-20T20:09:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4223 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T20:09:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:54Z Disabled control of system clock
I20260620 20:09:54.726635 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34277
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:09:55.043370 9410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:55.043635 9410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:55.043795 9410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:55.050971 9410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:55.051115 9410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:55.051190 9410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:55.051299 9410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:55.051390 9410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:09:55.061004 9410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:55.062482 9410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:55.064356 9410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:55.076093 9415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:55.076063 9416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:55.076129 9418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:55.076773 9410 server_base.cc:1034] running on GCE node
I20260620 20:09:55.077564 9410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:55.079067 9410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:55.080298 9410 hybrid_clock.cc:648] HybridClock initialized: now 1781986195080246 us; error 45 us; skew 500 ppm
I20260620 20:09:55.080649 9410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:09:55.083037 9410 webserver.cc:460] Webserver started at http://127.4.169.190:46439/ using document root <none> and password file <none>
I20260620 20:09:55.083739 9410 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:55.083878 9410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:55.084195 9410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:55.087085 9410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "d0db00bd76314f858f7ba47660984f94"
format_stamp: "Formatted at 2026-06-20 20:09:55 on dist-test-slave-4bwq"
server_key: "6368f2abcd483de64bad4756d894470f"
server_key_iv: "3c9d3101fe34e9f998a273b31c89bc7e"
server_key_version: "encryptionkey@0"
I20260620 20:09:55.087885 9410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "d0db00bd76314f858f7ba47660984f94"
format_stamp: "Formatted at 2026-06-20 20:09:55 on dist-test-slave-4bwq"
server_key: "6368f2abcd483de64bad4756d894470f"
server_key_iv: "3c9d3101fe34e9f998a273b31c89bc7e"
server_key_version: "encryptionkey@0"
I20260620 20:09:55.092991 9410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 20:09:55.096410 9424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:55.097707 9410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:09:55.097901 9410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "d0db00bd76314f858f7ba47660984f94"
format_stamp: "Formatted at 2026-06-20 20:09:55 on dist-test-slave-4bwq"
server_key: "6368f2abcd483de64bad4756d894470f"
server_key_iv: "3c9d3101fe34e9f998a273b31c89bc7e"
server_key_version: "encryptionkey@0"
I20260620 20:09:55.098125 9410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:55.103981 9410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:55.104844 9410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:55.105185 9410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:55.124142 9410 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:34277
I20260620 20:09:55.124169 9475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:34277 every 8 connection(s)
I20260620 20:09:55.125442 9410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:55.129103 9476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:55.133832 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 9410
I20260620 20:09:55.134030 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:55.134738 4774 external_mini_cluster.cc:1428] Setting key 4942d881e76217cc61876d7cf2be6d25
I20260620 20:09:55.139967 9476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94: Bootstrap starting.
I20260620 20:09:55.143253 9476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:55.143417 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:55.136652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.4.169.190:34277)
0620 20:09:55.137910 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:55.137921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:55.137964 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:09:55.139620 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:55.139643 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:55.140040 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:55.140479 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:55.141582 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:55.141589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:55.141601 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:55.141613 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:55.141670 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:55.141686 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:55.141929 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:55.142061 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 20:09:55.142507 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":346,"threads_started":1}
I20260620 20:09:55.144623 9476 log.cc:824] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:55.147524 9476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94: No bootstrap required, opened a new log
I20260620 20:09:55.154510 9476 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0db00bd76314f858f7ba47660984f94" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34277 } }
I20260620 20:09:55.154860 9476 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:55.154950 9476 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0db00bd76314f858f7ba47660984f94, State: Initialized, Role: FOLLOWER
I20260620 20:09:55.155575 9476 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0db00bd76314f858f7ba47660984f94" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34277 } }
I20260620 20:09:55.155807 9476 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:55.155959 9476 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:55.156132 9476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:55.158190 9476 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0db00bd76314f858f7ba47660984f94" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34277 } }
I20260620 20:09:55.158746 9476 leader_election.cc:302] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0db00bd76314f858f7ba47660984f94; no voters:
I20260620 20:09:55.159437 9476 leader_election.cc:288] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:55.159570 9481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:55.160308 9481 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [term 1 LEADER]: Becoming Leader. State: Replica: d0db00bd76314f858f7ba47660984f94, State: Running, Role: LEADER
I20260620 20:09:55.160820 9481 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0db00bd76314f858f7ba47660984f94" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34277 } }
I20260620 20:09:55.161497 9476 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:55.163785 9483 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0db00bd76314f858f7ba47660984f94. Latest consensus state: current_term: 1 leader_uuid: "d0db00bd76314f858f7ba47660984f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0db00bd76314f858f7ba47660984f94" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34277 } } }
I20260620 20:09:55.164217 9483 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:55.163980 9482 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0db00bd76314f858f7ba47660984f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0db00bd76314f858f7ba47660984f94" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34277 } } }
I20260620 20:09:55.164400 9482 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:55.170342 9486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:55.174767 9486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:55.183390 9486 catalog_manager.cc:1347] Generated new cluster ID: 61dae340ed46490a8f795dfb1697663b
I20260620 20:09:55.183508 9486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:55.209113 9486 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:55.209618 9486 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:55.219748 9486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94: Generated new TSK 0
I20260620 20:09:55.220820 9486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:09:55.225801 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34277
--builtin_ntp_servers=127.4.169.148:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:09:55.543879 9500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:55.544139 9500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:55.544229 9500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:55.551283 9500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:55.551378 9500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:55.551483 9500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:09:55.560748 9500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:55.561964 9500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:55.563740 9500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:55.576392 9505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:55.576592 9506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:55.576611 9508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:55.577069 9500 server_base.cc:1034] running on GCE node
I20260620 20:09:55.577603 9500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:55.578456 9500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:55.579677 9500 hybrid_clock.cc:648] HybridClock initialized: now 1781986195579638 us; error 47 us; skew 500 ppm
I20260620 20:09:55.579985 9500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:09:55.582059 9500 webserver.cc:460] Webserver started at http://127.4.169.129:39427/ using document root <none> and password file <none>
I20260620 20:09:55.582779 9500 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:55.582902 9500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:55.583220 9500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:55.585870 9500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "b3da0db6392b46c38454ec47dfc145c7"
format_stamp: "Formatted at 2026-06-20 20:09:55 on dist-test-slave-4bwq"
server_key: "ab1aa4c03f7e8d9fb75e08e6aa577099"
server_key_iv: "b38c1bd5ecb3e1c1343aa62301f08517"
server_key_version: "encryptionkey@0"
I20260620 20:09:55.586673 9500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "b3da0db6392b46c38454ec47dfc145c7"
format_stamp: "Formatted at 2026-06-20 20:09:55 on dist-test-slave-4bwq"
server_key: "ab1aa4c03f7e8d9fb75e08e6aa577099"
server_key_iv: "b38c1bd5ecb3e1c1343aa62301f08517"
server_key_version: "encryptionkey@0"
I20260620 20:09:55.591382 9500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 20:09:55.595218 9514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:55.596598 9500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:55.596776 9500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "b3da0db6392b46c38454ec47dfc145c7"
format_stamp: "Formatted at 2026-06-20 20:09:55 on dist-test-slave-4bwq"
server_key: "ab1aa4c03f7e8d9fb75e08e6aa577099"
server_key_iv: "b38c1bd5ecb3e1c1343aa62301f08517"
server_key_version: "encryptionkey@0"
I20260620 20:09:55.596997 9500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:55.602356 9500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:55.603147 9500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:55.603437 9500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:55.604288 9500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:55.605554 9500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:55.605674 9500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:55.605806 9500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:55.605892 9500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:55.638322 9500 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:43783
I20260620 20:09:55.638446 9626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:43783 every 8 connection(s)
I20260620 20:09:55.639600 9500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:09:55.644191 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 9500
I20260620 20:09:55.644404 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:09:55.644994 4774 external_mini_cluster.cc:1428] Setting key 81308eea1554a7b59d7422cc807d5ab3
I20260620 20:09:55.647667 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34277
--builtin_ntp_servers=127.4.169.148:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 20:09:55.647836 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:55.642086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:36777 (local address 127.4.169.190:34277)
0620 20:09:55.642302 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:55.642310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:55.643676 (+ 1366us) server_negotiation.cc:407] Connection header received
0620 20:09:55.645468 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:55.645474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:55.645542 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:55.645658 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:55.647131 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:55.647136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:55.647141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:55.647157 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:55.647196 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:55.647208 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:55.647330 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:55.647522 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 20:09:55.647582 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:09:55.648698 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:55.642675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34277 (local address 127.4.169.129:36777)
0620 20:09:55.643530 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:55.643565 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:55.645215 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:55.645906 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:55.645918 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:55.646036 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:55.646805 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:09:55.646809 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:55.646857 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:09:55.646966 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:55.647297 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:55.647308 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:55.647318 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:09:55.647589 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 20:09:55.647821 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":244,"threads_started":1}
I20260620 20:09:55.650067 9627 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34277
I20260620 20:09:55.650358 9627 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:55.650913 9627 heartbeater.cc:511] Master 127.4.169.190:34277 requested a full tablet report, sending...
I20260620 20:09:55.652870 9441 ts_manager.cc:194] Registered new tserver with Master: b3da0db6392b46c38454ec47dfc145c7 (127.4.169.129:43783)
W20260620 20:09:55.973363 9630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:55.973618 9630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:55.973762 9630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:55.980953 9630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:55.981050 9630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:55.981262 9630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:09:55.990144 9630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:55.991436 9630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:55.993188 9630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:56.006572 9635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:56.006515 9636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:56.006543 9638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:56.007474 9630 server_base.cc:1034] running on GCE node
I20260620 20:09:56.008100 9630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:56.009061 9630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:56.010319 9630 hybrid_clock.cc:648] HybridClock initialized: now 1781986196010247 us; error 60 us; skew 500 ppm
I20260620 20:09:56.010694 9630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:09:56.013017 9630 webserver.cc:460] Webserver started at http://127.4.169.130:45071/ using document root <none> and password file <none>
I20260620 20:09:56.013752 9630 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:56.013875 9630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:56.014213 9630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:56.017244 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "ff28fd6400264e57bc620f96d3d2fcc7"
format_stamp: "Formatted at 2026-06-20 20:09:56 on dist-test-slave-4bwq"
server_key: "d11dd447b9d2b5325043bf42727290e9"
server_key_iv: "270106dc9845f08749e477dfaf045ab2"
server_key_version: "encryptionkey@0"
I20260620 20:09:56.018026 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "ff28fd6400264e57bc620f96d3d2fcc7"
format_stamp: "Formatted at 2026-06-20 20:09:56 on dist-test-slave-4bwq"
server_key: "d11dd447b9d2b5325043bf42727290e9"
server_key_iv: "270106dc9845f08749e477dfaf045ab2"
server_key_version: "encryptionkey@0"
I20260620 20:09:56.023047 9630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 20:09:56.026278 9644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:56.027552 9630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:09:56.027732 9630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "ff28fd6400264e57bc620f96d3d2fcc7"
format_stamp: "Formatted at 2026-06-20 20:09:56 on dist-test-slave-4bwq"
server_key: "d11dd447b9d2b5325043bf42727290e9"
server_key_iv: "270106dc9845f08749e477dfaf045ab2"
server_key_version: "encryptionkey@0"
I20260620 20:09:56.027956 9630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:56.033663 9630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:56.034489 9630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:56.034762 9630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:56.035671 9630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:56.037019 9630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:56.037143 9630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:56.037284 9630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:56.037374 9630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:56.070288 9630 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:35555
I20260620 20:09:56.070358 9756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:35555 every 8 connection(s)
I20260620 20:09:56.071696 9630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:09:56.076608 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 9630
I20260620 20:09:56.076803 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:09:56.077239 4774 external_mini_cluster.cc:1428] Setting key fb37fe6d93f89f187a6995685858bac3
I20260620 20:09:56.079990 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.074208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:54777 (local address 127.4.169.190:34277)
0620 20:09:56.074468 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.074472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.075893 (+ 1421us) server_negotiation.cc:407] Connection header received
0620 20:09:56.077700 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.077707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.077776 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.077864 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.079345 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.079349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.079353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.079368 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.079400 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.079412 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.079524 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.079731 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.079779 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:09:56.080137 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34277
--builtin_ntp_servers=127.4.169.148:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 20:09:56.081328 9759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.074797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34277 (local address 127.4.169.130:54777)
0620 20:09:56.075740 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:56.075779 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:56.077435 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:56.078132 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:56.078141 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:56.078234 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:56.078987 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:09:56.078990 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:56.079048 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:09:56.079156 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:56.079517 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:56.079527 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:56.079535 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:09:56.079815 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 20:09:56.080072 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":280,"threads_started":1}
I20260620 20:09:56.085289 9757 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34277
I20260620 20:09:56.085592 9757 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:56.086171 9757 heartbeater.cc:511] Master 127.4.169.190:34277 requested a full tablet report, sending...
I20260620 20:09:56.087595 9441 ts_manager.cc:194] Registered new tserver with Master: ff28fd6400264e57bc620f96d3d2fcc7 (127.4.169.130:35555)
W20260620 20:09:56.402500 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:56.402832 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:56.402988 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:56.410228 9760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:56.410364 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:56.410586 9760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:09:56.419873 9760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:56.421252 9760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:56.423044 9760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:56.436062 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:56.436105 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:56.436148 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:56.436962 9760 server_base.cc:1034] running on GCE node
I20260620 20:09:56.437534 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:56.438474 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:56.439684 9760 hybrid_clock.cc:648] HybridClock initialized: now 1781986196439645 us; error 33 us; skew 500 ppm
I20260620 20:09:56.440014 9760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:09:56.442092 9760 webserver.cc:460] Webserver started at http://127.4.169.131:45105/ using document root <none> and password file <none>
I20260620 20:09:56.442806 9760 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:56.442916 9760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:56.443220 9760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:56.445905 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "277e0bf6f4b34f20b7bf80d916fcf41a"
format_stamp: "Formatted at 2026-06-20 20:09:56 on dist-test-slave-4bwq"
server_key: "46475a3baa86d16d0012fea1f697e6e2"
server_key_iv: "bc2a5f72c48996a81a5ac1f21d1e74e7"
server_key_version: "encryptionkey@0"
I20260620 20:09:56.446703 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "277e0bf6f4b34f20b7bf80d916fcf41a"
format_stamp: "Formatted at 2026-06-20 20:09:56 on dist-test-slave-4bwq"
server_key: "46475a3baa86d16d0012fea1f697e6e2"
server_key_iv: "bc2a5f72c48996a81a5ac1f21d1e74e7"
server_key_version: "encryptionkey@0"
I20260620 20:09:56.451566 9760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 20:09:56.455083 9774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:56.456435 9760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:09:56.456605 9760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "277e0bf6f4b34f20b7bf80d916fcf41a"
format_stamp: "Formatted at 2026-06-20 20:09:56 on dist-test-slave-4bwq"
server_key: "46475a3baa86d16d0012fea1f697e6e2"
server_key_iv: "bc2a5f72c48996a81a5ac1f21d1e74e7"
server_key_version: "encryptionkey@0"
I20260620 20:09:56.456845 9760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:56.463122 9760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:56.463905 9760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:56.464170 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:56.465070 9760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:09:56.466603 9760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:09:56.466691 9760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:56.466773 9760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:09:56.466828 9760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:56.497403 9760 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:40013
I20260620 20:09:56.497504 9886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:40013 every 8 connection(s)
I20260620 20:09:56.498638 9760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:09:56.506999 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.501289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:47779 (local address 127.4.169.190:34277)
0620 20:09:56.501467 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.501471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.503062 (+ 1591us) server_negotiation.cc:407] Connection header received
0620 20:09:56.504737 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.504743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.504822 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.504934 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.506258 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.506264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.506268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.506292 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.506322 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.506333 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.506481 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.506665 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.506733 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:09:56.507876 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.501872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34277 (local address 127.4.169.131:47779)
0620 20:09:56.502888 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:56.502939 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:56.504465 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:56.505171 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:56.505180 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:56.505272 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:56.505956 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:09:56.505959 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:56.505996 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:09:56.506076 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:56.506449 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:56.506459 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:56.506466 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:09:56.506755 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 20:09:56.506991 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260620 20:09:56.508453 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 9760
I20260620 20:09:56.508656 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:09:56.509213 4774 external_mini_cluster.cc:1428] Setting key 6c6d701180acfb472a38d48bdcbdccc8
I20260620 20:09:56.509351 9887 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34277
I20260620 20:09:56.509660 9887 heartbeater.cc:464] Registering TS with master...
I20260620 20:09:56.510362 9887 heartbeater.cc:511] Master 127.4.169.190:34277 requested a full tablet report, sending...
I20260620 20:09:56.511952 9440 ts_manager.cc:194] Registered new tserver with Master: 277e0bf6f4b34f20b7bf80d916fcf41a (127.4.169.131:40013)
I20260620 20:09:56.524309 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:09:56.530805 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.528532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.4.169.190:34277)
0620 20:09:56.528721 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.528729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.529139 (+ 410us) server_negotiation.cc:407] Connection header received
0620 20:09:56.529263 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.529266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.529311 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.529387 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.530159 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.530163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.530166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.530180 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.530207 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.530237 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.530330 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.530465 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.530510 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:09:56.534552 9440 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:09:56.541057 9440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:09:56.569494 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.552469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.4.169.129:43783)
0620 20:09:56.564666 (+ 12197us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.564674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.565378 (+ 704us) server_negotiation.cc:407] Connection header received
0620 20:09:56.565677 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.565685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.565925 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.566050 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.568579 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.568588 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.568600 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.568618 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.568676 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.568703 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.568850 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.569072 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.569171 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12035,"thread_start_us":11924,"threads_started":1}
I20260620 20:09:56.569551 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.553100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.4.169.131:40013)
0620 20:09:56.564019 (+ 10919us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.564044 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.565423 (+ 1379us) server_negotiation.cc:407] Connection header received
0620 20:09:56.565679 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.565686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.565923 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.566050 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.568452 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.568461 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.568475 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.568493 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.568553 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.568573 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.568738 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.569004 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.569109 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10684,"thread_start_us":10935,"threads_started":1}
I20260620 20:09:56.570659 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.552531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:40013 (local address 127.0.0.1:46338)
0620 20:09:56.565195 (+ 12664us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:56.565256 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:56.565512 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:56.567065 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:56.567072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:56.567191 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:56.568134 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:09:56.568139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:56.568175 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:09:56.568254 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:56.568797 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:56.568807 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:56.568814 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:09:56.569920 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0620 20:09:56.570024 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12546,"spinlock_wait_cycles":11136,"thread_start_us":12368,"threads_started":1}
I20260620 20:09:56.570703 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.552241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:43783 (local address 127.0.0.1:47052)
0620 20:09:56.565193 (+ 12952us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:56.565256 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:56.565512 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:56.567065 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:56.567072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:56.567191 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:56.568383 (+ 1192us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:09:56.568386 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:56.568402 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:09:56.568444 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:56.568902 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:56.568909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:56.568920 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:09:56.569973 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0620 20:09:56.570028 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12629,"spinlock_wait_cycles":768,"thread_start_us":12445,"threads_started":1}
I20260620 20:09:56.571691 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.566001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.4.169.130:35555)
0620 20:09:56.567975 (+ 1974us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.567985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.568114 (+ 129us) server_negotiation.cc:407] Connection header received
0620 20:09:56.568282 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.568292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.568522 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.568685 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.570898 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.570906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.570924 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.570945 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.571022 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.571046 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.571212 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.571288 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.571364 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466,"thread_start_us":1317,"threads_started":1}
I20260620 20:09:56.571722 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.565634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:35555 (local address 127.0.0.1:47760)
0620 20:09:56.567971 (+ 2337us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:56.568010 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:56.568122 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:56.569695 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:56.569699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:56.569724 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:56.570530 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:09:56.570534 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:56.570553 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:09:56.570602 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:56.571161 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:56.571166 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:56.571172 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:09:56.571289 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 20:09:56.571340 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2240,"thread_start_us":2068,"threads_started":1}
I20260620 20:09:56.574038 9562 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec4652e1854405c99ed2de891316467 (DEFAULT_TABLE table=test-table [id=5e4a01e1e6d94006aff7d2b3b22cd73a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:56.574060 9822 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec4652e1854405c99ed2de891316467 (DEFAULT_TABLE table=test-table [id=5e4a01e1e6d94006aff7d2b3b22cd73a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:56.575379 9822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec4652e1854405c99ed2de891316467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:56.577075 9692 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec4652e1854405c99ed2de891316467 (DEFAULT_TABLE table=test-table [id=5e4a01e1e6d94006aff7d2b3b22cd73a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:09:56.577616 9562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec4652e1854405c99ed2de891316467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:56.578128 9692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec4652e1854405c99ed2de891316467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:56.588254 9905 tablet_bootstrap.cc:492] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: Bootstrap starting.
I20260620 20:09:56.591516 9905 tablet_bootstrap.cc:654] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:56.592693 9905 log.cc:824] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:56.592909 9907 tablet_bootstrap.cc:492] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7: Bootstrap starting.
I20260620 20:09:56.594502 9906 tablet_bootstrap.cc:492] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: Bootstrap starting.
I20260620 20:09:56.597426 9907 tablet_bootstrap.cc:654] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:56.597616 9906 tablet_bootstrap.cc:654] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:56.598867 9906 log.cc:824] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:56.599095 9907 log.cc:824] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:56.602566 9905 tablet_bootstrap.cc:492] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: No bootstrap required, opened a new log
I20260620 20:09:56.602674 9907 tablet_bootstrap.cc:492] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7: No bootstrap required, opened a new log
I20260620 20:09:56.602851 9905 ts_tablet_manager.cc:1397] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260620 20:09:56.602872 9907 ts_tablet_manager.cc:1397] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 20:09:56.609429 9907 raft_consensus.cc:348] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } }
I20260620 20:09:56.609428 9905 raft_consensus.cc:348] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } }
I20260620 20:09:56.609779 9905 raft_consensus.cc:374] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:56.609794 9907 raft_consensus.cc:374] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:56.609862 9905 raft_consensus.cc:729] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff28fd6400264e57bc620f96d3d2fcc7, State: Initialized, Role: FOLLOWER
I20260620 20:09:56.609877 9907 raft_consensus.cc:729] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da0db6392b46c38454ec47dfc145c7, State: Initialized, Role: FOLLOWER
I20260620 20:09:56.610533 9907 consensus_queue.cc:260] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } }
I20260620 20:09:56.610538 9905 consensus_queue.cc:260] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } }
I20260620 20:09:56.611701 9627 heartbeater.cc:503] Master 127.4.169.190:34277 was elected leader, sending a full tablet report...
I20260620 20:09:56.612154 9757 heartbeater.cc:503] Master 127.4.169.190:34277 was elected leader, sending a full tablet report...
I20260620 20:09:56.612301 9907 ts_tablet_manager.cc:1428] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260620 20:09:56.612547 9906 tablet_bootstrap.cc:492] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: No bootstrap required, opened a new log
I20260620 20:09:56.612771 9906 ts_tablet_manager.cc:1397] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260620 20:09:56.614107 9905 ts_tablet_manager.cc:1428] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260620 20:09:56.621012 9906 raft_consensus.cc:348] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } }
I20260620 20:09:56.621320 9906 raft_consensus.cc:374] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:56.621395 9906 raft_consensus.cc:729] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277e0bf6f4b34f20b7bf80d916fcf41a, State: Initialized, Role: FOLLOWER
I20260620 20:09:56.621836 9906 consensus_queue.cc:260] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } }
I20260620 20:09:56.622766 9887 heartbeater.cc:503] Master 127.4.169.190:34277 was elected leader, sending a full tablet report...
I20260620 20:09:56.623178 9906 ts_tablet_manager.cc:1428] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
W20260620 20:09:56.641618 9628 tablet.cc:2369] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:09:56.750375 9888 tablet.cc:2369] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:56.779153 9911 raft_consensus.cc:482] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:09:56.779516 9911 raft_consensus.cc:504] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } }
I20260620 20:09:56.781245 9911 leader_election.cc:288] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 277e0bf6f4b34f20b7bf80d916fcf41a (127.4.169.131:40013), ff28fd6400264e57bc620f96d3d2fcc7 (127.4.169.130:35555)
I20260620 20:09:56.784662 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.781806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:53111 (local address 127.4.169.131:40013)
0620 20:09:56.782034 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.782041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.782348 (+ 307us) server_negotiation.cc:407] Connection header received
0620 20:09:56.782557 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.782563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.782666 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.782781 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.783936 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.783944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.783956 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.783980 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.784024 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.784043 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.784166 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.784316 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.784376 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:09:56.784983 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.782488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:60303 (local address 127.4.169.130:35555)
0620 20:09:56.782684 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.782692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.783095 (+ 403us) server_negotiation.cc:407] Connection header received
0620 20:09:56.783199 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.783203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.783286 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.783398 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.784364 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.784369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.784373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.784391 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.784428 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.784443 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.784567 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.784690 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.784748 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:09:56.785117 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.781610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:40013 (local address 127.4.169.129:53111)
0620 20:09:56.782195 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:56.782211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:56.782379 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:56.782827 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:56.782831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:56.782857 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:56.783690 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:09:56.783696 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:56.783716 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:09:56.783770 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:56.784169 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:56.784174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:56.784180 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:09:56.784343 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 20:09:56.784395 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":316,"threads_started":1}
I20260620 20:09:56.785380 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.782293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:35555 (local address 127.4.169.129:60303)
0620 20:09:56.782961 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:56.782983 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:56.783097 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:56.783446 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:56.783453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:56.783477 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:09:56.784170 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:09:56.784173 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:09:56.784186 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:09:56.784225 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:09:56.784549 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:09:56.784555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:09:56.784560 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:09:56.784701 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 20:09:56.784754 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":311,"threads_started":1}
I20260620 20:09:56.786295 9712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec4652e1854405c99ed2de891316467" candidate_uuid: "b3da0db6392b46c38454ec47dfc145c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" is_pre_election: true
I20260620 20:09:56.786295 9842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec4652e1854405c99ed2de891316467" candidate_uuid: "b3da0db6392b46c38454ec47dfc145c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" is_pre_election: true
I20260620 20:09:56.786900 9842 raft_consensus.cc:2454] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da0db6392b46c38454ec47dfc145c7 in term 0.
I20260620 20:09:56.786900 9712 raft_consensus.cc:2454] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da0db6392b46c38454ec47dfc145c7 in term 0.
I20260620 20:09:56.787591 9515 leader_election.cc:302] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277e0bf6f4b34f20b7bf80d916fcf41a, b3da0db6392b46c38454ec47dfc145c7; no voters:
I20260620 20:09:56.788053 9911 raft_consensus.cc:2781] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:09:56.788215 9911 raft_consensus.cc:482] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:09:56.788340 9911 raft_consensus.cc:3037] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:56.790536 9911 raft_consensus.cc:504] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } }
I20260620 20:09:56.791085 9911 leader_election.cc:288] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [CANDIDATE]: Term 1 election: Requested vote from peers 277e0bf6f4b34f20b7bf80d916fcf41a (127.4.169.131:40013), ff28fd6400264e57bc620f96d3d2fcc7 (127.4.169.130:35555)
I20260620 20:09:56.791608 9842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec4652e1854405c99ed2de891316467" candidate_uuid: "b3da0db6392b46c38454ec47dfc145c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a"
I20260620 20:09:56.791636 9712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec4652e1854405c99ed2de891316467" candidate_uuid: "b3da0db6392b46c38454ec47dfc145c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28fd6400264e57bc620f96d3d2fcc7"
I20260620 20:09:56.791822 9842 raft_consensus.cc:3037] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:56.791822 9712 raft_consensus.cc:3037] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:56.795037 9712 raft_consensus.cc:2454] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da0db6392b46c38454ec47dfc145c7 in term 1.
I20260620 20:09:56.795037 9842 raft_consensus.cc:2454] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da0db6392b46c38454ec47dfc145c7 in term 1.
I20260620 20:09:56.795389 9515 leader_election.cc:302] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277e0bf6f4b34f20b7bf80d916fcf41a, b3da0db6392b46c38454ec47dfc145c7; no voters:
I20260620 20:09:56.795675 9911 raft_consensus.cc:2781] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:56.796082 9911 raft_consensus.cc:686] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 1 LEADER]: Becoming Leader. State: Replica: b3da0db6392b46c38454ec47dfc145c7, State: Running, Role: LEADER
I20260620 20:09:56.796530 9911 consensus_queue.cc:237] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } }
I20260620 20:09:56.800958 9441 catalog_manager.cc:5495] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da0db6392b46c38454ec47dfc145c7 (127.4.169.129). New cstate: current_term: 1 leader_uuid: "b3da0db6392b46c38454ec47dfc145c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da0db6392b46c38454ec47dfc145c7" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 43783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 } health_report { overall_health: UNKNOWN } } }
W20260620 20:09:56.823678 9758 tablet.cc:2369] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:09:56.853572 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.851412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.4.169.129:43783)
0620 20:09:56.851612 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.851618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.851639 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:09:56.851853 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.851857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.851935 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.852047 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.852959 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.852965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.852969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.852986 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.853027 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.853038 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.853164 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.853244 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.853302 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:09:56.861685 9712 raft_consensus.cc:1261] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [term 1 FOLLOWER]: Refusing update from remote peer b3da0db6392b46c38454ec47dfc145c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:56.861686 9842 raft_consensus.cc:1261] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [term 1 FOLLOWER]: Refusing update from remote peer b3da0db6392b46c38454ec47dfc145c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:09:56.862629 9916 consensus_queue.cc:1035] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "277e0bf6f4b34f20b7bf80d916fcf41a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:56.862962 9911 consensus_queue.cc:1035] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff28fd6400264e57bc620f96d3d2fcc7" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:09:56.872987 9921 mvcc.cc:205] Tried to move back new op lower bound from 7299015462335303680 to 7299015462082232320. Current Snapshot: MvccSnapshot[applied={T|T < 7299015462335303680}]
I20260620 20:09:56.883190 9920 mvcc.cc:205] Tried to move back new op lower bound from 7299015462335303680 to 7299015462082232320. Current Snapshot: MvccSnapshot[applied={T|T < 7299015462335303680}]
I20260620 20:09:56.891207 9441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:56.891467 9441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:09:56.894972 9441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0db00bd76314f858f7ba47660984f94: Sending DeleteTablet for 3 replicas of tablet 5ec4652e1854405c99ed2de891316467
I20260620 20:09:56.896135 9562 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec4652e1854405c99ed2de891316467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:56 UTC) from {username='slave'} at 127.0.0.1:47052
I20260620 20:09:56.896417 9822 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec4652e1854405c99ed2de891316467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:56 UTC) from {username='slave'} at 127.0.0.1:46338
I20260620 20:09:56.896417 9692 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec4652e1854405c99ed2de891316467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:09:56 UTC) from {username='slave'} at 127.0.0.1:47760
I20260620 20:09:56.898533 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 9500
I20260620 20:09:56.907796 9930 tablet_replica.cc:330] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: stopping tablet replica
I20260620 20:09:56.909060 9930 raft_consensus.cc:2229] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:56.910404 9930 raft_consensus.cc:2258] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:56.911373 9931 tablet_replica.cc:330] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: stopping tablet replica
I20260620 20:09:56.912310 9931 raft_consensus.cc:2229] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:09:56.914533 9930 ts_tablet_manager.cc:1905] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:56.914750 9929 tablet_replica.cc:330] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7: stopping tablet replica
I20260620 20:09:56.915127 9931 raft_consensus.cc:2258] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:56.915112 9929 raft_consensus.cc:2229] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:09:56.915769 9929 raft_consensus.cc:2258] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:09:56.915860 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:56.900033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.4.169.129:43783)
0620 20:09:56.900224 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:56.900233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:56.913022 (+ 12789us) server_negotiation.cc:407] Connection header received
0620 20:09:56.913532 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:56.913542 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:56.913610 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:56.913688 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:56.914974 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:56.914980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:56.914985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:56.915011 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:56.915046 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:56.915061 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:56.915183 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:56.915454 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 20:09:56.915516 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:09:56.916970 9622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:56.917795 9931 ts_tablet_manager.cc:1905] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:56.924469 9930 ts_tablet_manager.cc:1918] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:09:56.924646 9930 log.cc:1196] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/5ec4652e1854405c99ed2de891316467
I20260620 20:09:56.925092 9930 ts_tablet_manager.cc:1939] T 5ec4652e1854405c99ed2de891316467 P 277e0bf6f4b34f20b7bf80d916fcf41a: Deleting consensus metadata
I20260620 20:09:56.925901 9931 ts_tablet_manager.cc:1918] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:09:56.926060 9931 log.cc:1196] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/5ec4652e1854405c99ed2de891316467
I20260620 20:09:56.926565 9931 ts_tablet_manager.cc:1939] T 5ec4652e1854405c99ed2de891316467 P ff28fd6400264e57bc620f96d3d2fcc7: Deleting consensus metadata
I20260620 20:09:56.926700 9425 catalog_manager.cc:4841] TS 277e0bf6f4b34f20b7bf80d916fcf41a (127.4.169.131:40013): tablet 5ec4652e1854405c99ed2de891316467 (table test-table [id=5e4a01e1e6d94006aff7d2b3b22cd73a]) successfully deleted
I20260620 20:09:56.928216 9425 catalog_manager.cc:4841] TS ff28fd6400264e57bc620f96d3d2fcc7 (127.4.169.130:35555): tablet 5ec4652e1854405c99ed2de891316467 (table test-table [id=5e4a01e1e6d94006aff7d2b3b22cd73a]) successfully deleted
I20260620 20:09:56.917793 9929 ts_tablet_manager.cc:1905] T 5ec4652e1854405c99ed2de891316467 P b3da0db6392b46c38454ec47dfc145c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:09:57.012763 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 9500
W20260620 20:09:57.026320 9428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.169.129:43783 (error 108)
I20260620 20:09:57.027024 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 9630
I20260620 20:09:57.028291 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.027735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:43783 (local address 127.0.0.1:47082)
0620 20:09:57.027930 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:57.028090 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 20:09:57.028635 9428 catalog_manager.cc:4584] TS b3da0db6392b46c38454ec47dfc145c7 (127.4.169.129:43783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ec4652e1854405c99ed2de891316467: Network error: Client connection negotiation failed: client connection to 127.4.169.129:43783: connect: Connection refused (error 111)
I20260620 20:09:57.030467 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.027915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.4.169.130:35555)
0620 20:09:57.028101 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:57.028107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:57.028128 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:09:57.028284 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:57.028291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:57.028401 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:57.028507 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:57.029674 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:57.029680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:57.029687 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:57.029706 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:57.029758 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:57.029780 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:57.029943 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:57.030063 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 20:09:57.030114 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:09:57.033201 9752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:57.082130 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.081684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:43783 (local address 127.0.0.1:47084)
0620 20:09:57.081881 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:57.081990 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 20:09:57.083108 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.082814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:43783 (local address 127.0.0.1:47098)
0620 20:09:57.082938 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:57.083004 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 20:09:57.122017 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 9630
I20260620 20:09:57.142954 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 9760
I20260620 20:09:57.143788 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.143238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:43783 (local address 127.0.0.1:47114)
0620 20:09:57.143427 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:57.143561 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:09:57.144881 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.144532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:43783 (local address 127.0.0.1:47128)
0620 20:09:57.144695 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:57.144770 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 20:09:57.146302 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.143803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.4.169.131:40013)
0620 20:09:57.144043 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:57.144050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:57.144326 (+ 276us) server_negotiation.cc:407] Connection header received
0620 20:09:57.144522 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:57.144527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:57.144630 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:57.144725 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:57.145742 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:57.145747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:57.145751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:57.145765 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:57.145798 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:57.145809 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:57.145914 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:57.146017 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 20:09:57.146062 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 20:09:57.146992 9882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:57.225721 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.225126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:43783 (local address 127.0.0.1:47140)
0620 20:09:57.225458 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:57.225562 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260620 20:09:57.226846 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.226459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:43783 (local address 127.0.0.1:47150)
0620 20:09:57.226653 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:57.226727 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 20:09:57.234467 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 9760
I20260620 20:09:57.250061 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 9410
I20260620 20:09:57.251201 9471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:57.354916 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 9410
2026-06-20T20:09:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2694 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 20:09:57.395978 4774 mini_oidc.cc:152] Starting JWKS server
I20260620 20:09:57.398517 4774 webserver.cc:460] Webserver started at https://127.0.0.1:44835/ using document root <none> and password file <none>
I20260620 20:09:57.398780 4774 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:09:57.399407 4774 webserver.cc:460] Webserver started at http://127.0.0.1:38517/ using document root <none> and password file <none>
2026-06-20T20:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:57Z Disabled control of system clock
I20260620 20:09:57.426301 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:40815
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:38991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:40815
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44835/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/testchainca.cert with env {}
W20260620 20:09:57.743638 9943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:57.743929 9943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:57.744262 9943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:57.751675 9943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:57.751785 9943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:57.751881 9943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:57.751982 9943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:57.752063 9943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:09:57.752159 9943 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:09:57.752233 9943 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44835/jwks.json
I20260620 20:09:57.762454 9943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:40815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:40815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44835/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:57.763794 9943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:57.765466 9943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:57.777387 9948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:57.777416 9949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:57.777352 9951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:57.778503 9943 server_base.cc:1034] running on GCE node
I20260620 20:09:57.779325 9943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:57.780712 9943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:57.781942 9943 hybrid_clock.cc:648] HybridClock initialized: now 1781986197781904 us; error 53 us; skew 500 ppm
I20260620 20:09:57.782294 9943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:09:57.784498 9943 webserver.cc:460] Webserver started at http://127.4.169.190:44189/ using document root <none> and password file <none>
I20260620 20:09:57.785189 9943 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:57.785334 9943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:57.785642 9943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:57.788578 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "3a026440208a418283b82ab5cddf48fb"
format_stamp: "Formatted at 2026-06-20 20:09:57 on dist-test-slave-4bwq"
server_key: "e5a1bbe9cb1993e6cd4b7bddbd803017"
server_key_iv: "332a8492a73de41111710dc87884f0ee"
server_key_version: "encryptionkey@0"
I20260620 20:09:57.789364 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "3a026440208a418283b82ab5cddf48fb"
format_stamp: "Formatted at 2026-06-20 20:09:57 on dist-test-slave-4bwq"
server_key: "e5a1bbe9cb1993e6cd4b7bddbd803017"
server_key_iv: "332a8492a73de41111710dc87884f0ee"
server_key_version: "encryptionkey@0"
I20260620 20:09:57.794263 9943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:09:57.797552 9957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:57.798926 9943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:09:57.799096 9943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "3a026440208a418283b82ab5cddf48fb"
format_stamp: "Formatted at 2026-06-20 20:09:57 on dist-test-slave-4bwq"
server_key: "e5a1bbe9cb1993e6cd4b7bddbd803017"
server_key_iv: "332a8492a73de41111710dc87884f0ee"
server_key_version: "encryptionkey@0"
I20260620 20:09:57.799329 9943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:57.815130 9943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:57.815879 9943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:57.816188 9943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:57.834316 9943 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:40815
I20260620 20:09:57.834334 10008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:40815 every 8 connection(s)
I20260620 20:09:57.835703 9943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:57.839313 10009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:57.842782 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 9943
I20260620 20:09:57.842965 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:57.843763 4774 external_mini_cluster.cc:1428] Setting key cf8b91c3e133b9cce76151f797aa1a3d
I20260620 20:09:57.851545 10009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb: Bootstrap starting.
I20260620 20:09:57.854720 10009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:57.855854 10009 log.cc:824] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:57.857095 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.845668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.4.169.190:40815)
0620 20:09:57.846630 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:57.846641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:57.846680 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:09:57.848307 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:57.848328 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:57.848782 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:57.849162 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:57.849767 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:57.851065 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:57.852443 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:57.853116 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:57.853333 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:57.854529 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:57.854536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:57.854548 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:57.855117 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:57.855231 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:57.855254 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:57.855381 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:57.855601 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 20:09:57.855880 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":286,"threads_started":1}
I20260620 20:09:57.858929 10009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb: No bootstrap required, opened a new log
I20260620 20:09:57.860819 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:09:57.867072 10009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a026440208a418283b82ab5cddf48fb" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40815 } }
I20260620 20:09:57.867477 10009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:57.867585 10009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a026440208a418283b82ab5cddf48fb, State: Initialized, Role: FOLLOWER
I20260620 20:09:57.868187 10009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a026440208a418283b82ab5cddf48fb" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40815 } }
I20260620 20:09:57.868438 10009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:57.868588 10009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:57.868770 10009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:57.870959 10009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a026440208a418283b82ab5cddf48fb" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40815 } }
I20260620 20:09:57.871397 10009 leader_election.cc:302] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a026440208a418283b82ab5cddf48fb; no voters:
I20260620 20:09:57.871953 10009 leader_election.cc:288] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:57.872097 10014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:57.872766 10014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [term 1 LEADER]: Becoming Leader. State: Replica: 3a026440208a418283b82ab5cddf48fb, State: Running, Role: LEADER
I20260620 20:09:57.873235 10014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a026440208a418283b82ab5cddf48fb" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40815 } }
I20260620 20:09:57.873986 10009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:57.876165 10015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a026440208a418283b82ab5cddf48fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a026440208a418283b82ab5cddf48fb" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40815 } } }
I20260620 20:09:57.876647 10015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:57.876194 10016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a026440208a418283b82ab5cddf48fb. Latest consensus state: current_term: 1 leader_uuid: "3a026440208a418283b82ab5cddf48fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a026440208a418283b82ab5cddf48fb" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40815 } } }
I20260620 20:09:57.877065 10016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:57.877681 10022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:57.885880 10022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:57.894586 10022 catalog_manager.cc:1347] Generated new cluster ID: 5a9c1d047384493398f5d53368cda5d6
I20260620 20:09:57.894697 10022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:57.911150 10022 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:57.912777 10022 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:57.925652 10022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a026440208a418283b82ab5cddf48fb: Generated new TSK 0
I20260620 20:09:57.927037 10022 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:09:57.973706 10039 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 20:09:57.973953 10039 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:09:57.975215 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:57.951897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.4.169.190:40815)
0620 20:09:57.952105 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:57.952113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:57.952779 (+ 666us) server_negotiation.cc:407] Connection header received
0620 20:09:57.953120 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:57.953128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:57.953249 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:57.953428 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:09:57.953924 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:57.955115 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:57.956604 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:57.957296 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:57.957431 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:57.957997 (+ 566us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:09:57.974479 (+ 16482us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:09:57.974875 (+ 396us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 20:09:57.975344 10012 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 20:09:57.976928 10038 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:09:57.951903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40815 (local address 127.0.0.1:54388)
0620 20:09:57.952668 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:57.952685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:57.952848 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:57.953462 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:57.953466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:57.953476 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:09:57.953785 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:57.953793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:57.955295 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:57.955299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:57.956459 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:57.956471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:57.957515 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:57.957520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:57.957651 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:57.957716 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:09:57.974959 (+ 17243us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:09:57.975448 (+ 489us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.169.190:40815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":411,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.169.190:40815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44835/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 20:09:57.978953 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 9943
I20260620 20:09:57.980270 10004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:58.074007 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 9943
2026-06-20T20:09:58Z chronyd exiting
I20260620 20:09:58.124701 4774 test_util.cc:175] -----------------------------------------------
I20260620 20:09:58.124876 4774 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986152907614-4774-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (733 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 20:09:58.128772 4774 mini_oidc.cc:152] Starting JWKS server
I20260620 20:09:58.130613 4774 webserver.cc:460] Webserver started at https://127.0.0.1:40393/ using document root <none> and password file <none>
I20260620 20:09:58.130864 4774 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:09:58.131476 4774 webserver.cc:460] Webserver started at http://127.0.0.1:45575/ using document root <none> and password file <none>
2026-06-20T20:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:58Z Disabled control of system clock
I20260620 20:09:58.158110 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:44557
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:44557
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40393/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/testchainca.cert with env {}
W20260620 20:09:58.473557 10046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:58.473934 10046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:58.474120 10046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:58.481609 10046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:58.481786 10046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:58.481882 10046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:58.481988 10046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:58.482075 10046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:09:58.482165 10046 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:09:58.482255 10046 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40393/jwks.json
I20260620 20:09:58.492715 10046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:44557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:44557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40393/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:58.494086 10046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:58.495468 10046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:58.507457 10051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:58.507366 10052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:58.507526 10054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:58.508548 10046 server_base.cc:1034] running on GCE node
I20260620 20:09:58.509372 10046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:58.510838 10046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:58.512033 10046 hybrid_clock.cc:648] HybridClock initialized: now 1781986198511993 us; error 28 us; skew 500 ppm
I20260620 20:09:58.512391 10046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:09:58.518441 10046 webserver.cc:460] Webserver started at http://127.4.169.190:41733/ using document root <none> and password file <none>
I20260620 20:09:58.519174 10046 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:58.519299 10046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:58.519631 10046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:58.522495 10046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "67411beb938b4a45909c838ee2ff2d8a"
format_stamp: "Formatted at 2026-06-20 20:09:58 on dist-test-slave-4bwq"
server_key: "4a144b477ec2de196ceabccdc66cde12"
server_key_iv: "89397b7c53b5312773e7d44c15ee3c43"
server_key_version: "encryptionkey@0"
I20260620 20:09:58.523283 10046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "67411beb938b4a45909c838ee2ff2d8a"
format_stamp: "Formatted at 2026-06-20 20:09:58 on dist-test-slave-4bwq"
server_key: "4a144b477ec2de196ceabccdc66cde12"
server_key_iv: "89397b7c53b5312773e7d44c15ee3c43"
server_key_version: "encryptionkey@0"
I20260620 20:09:58.528275 10046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 20:09:58.531605 10060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:58.532893 10046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:09:58.533070 10046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "67411beb938b4a45909c838ee2ff2d8a"
format_stamp: "Formatted at 2026-06-20 20:09:58 on dist-test-slave-4bwq"
server_key: "4a144b477ec2de196ceabccdc66cde12"
server_key_iv: "89397b7c53b5312773e7d44c15ee3c43"
server_key_version: "encryptionkey@0"
I20260620 20:09:58.533282 10046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:58.549940 10046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:58.550774 10046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:58.551087 10046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:58.569104 10046 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:44557
I20260620 20:09:58.569111 10111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:44557 every 8 connection(s)
I20260620 20:09:58.570468 10046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:58.574193 10112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:58.575455 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 10046
I20260620 20:09:58.575662 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:58.576299 4774 external_mini_cluster.cc:1428] Setting key 603e616d54e8f43346c096e7ec46f438
I20260620 20:09:58.588500 10112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a: Bootstrap starting.
I20260620 20:09:58.591014 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:58.578229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.4.169.190:44557)
0620 20:09:58.579367 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:58.579381 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:58.579432 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:09:58.581073 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:58.581099 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:58.581713 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:58.582149 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:58.582661 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:58.584528 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:58.585739 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:58.586813 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:58.586990 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:58.588172 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:58.588184 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:58.588201 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:58.588868 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:58.588969 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:58.588997 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:58.589114 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:58.589326 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 20:09:58.589646 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":385,"threads_started":1}
I20260620 20:09:58.592283 10112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:58.593391 10112 log.cc:824] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:58.595110 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:09:58.596925 10112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a: No bootstrap required, opened a new log
I20260620 20:09:58.605820 10112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67411beb938b4a45909c838ee2ff2d8a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 44557 } }
I20260620 20:09:58.606333 10112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:58.606459 10112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67411beb938b4a45909c838ee2ff2d8a, State: Initialized, Role: FOLLOWER
I20260620 20:09:58.607048 10112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67411beb938b4a45909c838ee2ff2d8a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 44557 } }
I20260620 20:09:58.607291 10112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:58.607390 10112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:58.607507 10112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:58.609977 10112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67411beb938b4a45909c838ee2ff2d8a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 44557 } }
I20260620 20:09:58.610508 10112 leader_election.cc:302] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67411beb938b4a45909c838ee2ff2d8a; no voters:
I20260620 20:09:58.611018 10112 leader_election.cc:288] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:58.611191 10117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:58.611928 10117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [term 1 LEADER]: Becoming Leader. State: Replica: 67411beb938b4a45909c838ee2ff2d8a, State: Running, Role: LEADER
I20260620 20:09:58.612448 10117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67411beb938b4a45909c838ee2ff2d8a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 44557 } }
I20260620 20:09:58.613243 10112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:58.615423 10119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67411beb938b4a45909c838ee2ff2d8a. Latest consensus state: current_term: 1 leader_uuid: "67411beb938b4a45909c838ee2ff2d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67411beb938b4a45909c838ee2ff2d8a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 44557 } } }
I20260620 20:09:58.615450 10118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67411beb938b4a45909c838ee2ff2d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67411beb938b4a45909c838ee2ff2d8a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 44557 } } }
I20260620 20:09:58.615864 10118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:58.615864 10119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:58.617528 10122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:58.622658 10122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:58.631943 10122 catalog_manager.cc:1347] Generated new cluster ID: 537a3c9968074854864ff732e477aea5
I20260620 20:09:58.632053 10122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:58.653707 10122 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:58.655666 10122 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:58.670835 10122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67411beb938b4a45909c838ee2ff2d8a: Generated new TSK 0
I20260620 20:09:58.671960 10122 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:09:58.711655 10142 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 20:09:58.711787 10142 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:09:58.713110 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:58.692884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.4.169.190:44557)
0620 20:09:58.693079 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:58.693085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:58.693427 (+ 342us) server_negotiation.cc:407] Connection header received
0620 20:09:58.693616 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:58.693623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:58.693714 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:58.693880 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:09:58.694276 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:58.695420 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:58.697074 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:58.697768 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:58.697913 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:58.698335 (+ 422us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:09:58.712369 (+ 14034us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:09:58.712786 (+ 417us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:09:58.713177 10141 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:09:58.692727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:44557 (local address 127.0.0.1:38932)
0620 20:09:58.693330 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:58.693346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:58.693460 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:58.693895 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:58.693899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:58.693907 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:09:58.694126 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:58.694133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:58.695616 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:58.695621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:58.696907 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:58.696920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:58.697924 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:58.697929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:58.698059 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:58.698090 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:09:58.712622 (+ 14532us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:09:58.712820 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.169.190:44557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":323,"threads_started":1}
W20260620 20:09:58.713253 10115 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 20:09:58.714274 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 10046
I20260620 20:09:58.715523 10107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:09:58.807444 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 10046
2026-06-20T20:09:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (733 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 20:09:58.862653 4774 mini_oidc.cc:152] Starting JWKS server
I20260620 20:09:58.864487 4774 webserver.cc:460] Webserver started at https://127.0.0.1:35311/ using document root <none> and password file <none>
I20260620 20:09:58.864722 4774 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:09:58.865309 4774 webserver.cc:460] Webserver started at http://127.0.0.1:37517/ using document root <none> and password file <none>
2026-06-20T20:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:58Z Disabled control of system clock
I20260620 20:09:58.891352 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:32997
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:32997
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35311/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:09:59.223981 10149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:59.224371 10149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:59.224550 10149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:59.231930 10149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:59.232093 10149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:59.232165 10149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:59.232262 10149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:59.232328 10149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:09:59.232421 10149 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:09:59.232493 10149 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35311/jwks.json
I20260620 20:09:59.241629 10149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:32997
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35311/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:59.243005 10149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:59.244793 10149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:59.258714 10157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:59.258790 10154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:59.258816 10155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:59.259825 10149 server_base.cc:1034] running on GCE node
I20260620 20:09:59.260500 10149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:59.261781 10149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:59.262970 10149 hybrid_clock.cc:648] HybridClock initialized: now 1781986199262926 us; error 41 us; skew 500 ppm
I20260620 20:09:59.263269 10149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:09:59.265475 10149 webserver.cc:460] Webserver started at http://127.4.169.190:37841/ using document root <none> and password file <none>
I20260620 20:09:59.266108 10149 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:59.266191 10149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:59.266511 10149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:59.269375 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "491616eb440643b39a0998f41adf1a68"
format_stamp: "Formatted at 2026-06-20 20:09:59 on dist-test-slave-4bwq"
server_key: "7750164ab3f02f4a3894a1104b1ec7d6"
server_key_iv: "0d161493e9dd8d518d7fd789ba60c81f"
server_key_version: "encryptionkey@0"
I20260620 20:09:59.270097 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "491616eb440643b39a0998f41adf1a68"
format_stamp: "Formatted at 2026-06-20 20:09:59 on dist-test-slave-4bwq"
server_key: "7750164ab3f02f4a3894a1104b1ec7d6"
server_key_iv: "0d161493e9dd8d518d7fd789ba60c81f"
server_key_version: "encryptionkey@0"
I20260620 20:09:59.274946 10149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:09:59.278028 10163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:59.279376 10149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:09:59.279556 10149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "491616eb440643b39a0998f41adf1a68"
format_stamp: "Formatted at 2026-06-20 20:09:59 on dist-test-slave-4bwq"
server_key: "7750164ab3f02f4a3894a1104b1ec7d6"
server_key_iv: "0d161493e9dd8d518d7fd789ba60c81f"
server_key_version: "encryptionkey@0"
I20260620 20:09:59.279771 10149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:59.296803 10149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:59.297610 10149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:09:59.297926 10149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:09:59.315758 10149 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:32997
I20260620 20:09:59.315784 10214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:32997 every 8 connection(s)
I20260620 20:09:59.317090 10149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:09:59.320748 10215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:09:59.324043 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 10149
I20260620 20:09:59.324234 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:09:59.325043 4774 external_mini_cluster.cc:1428] Setting key 5d7a3c6099da056012be8b3a6134edfc
I20260620 20:09:59.333320 10215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68: Bootstrap starting.
I20260620 20:09:59.336951 10215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68: Neither blocks nor log segments found. Creating new log.
I20260620 20:09:59.338033 10215 log.cc:824] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68: Log is configured to *not* fsync() on all Append() calls
I20260620 20:09:59.338263 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:59.326807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.4.169.190:32997)
0620 20:09:59.327789 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:59.327800 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:59.327840 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:09:59.329461 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:59.329480 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:59.329936 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:59.330263 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:09:59.330830 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:59.332106 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:59.333936 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:59.334651 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:59.334783 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:59.335991 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:09:59.335999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:09:59.336011 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:09:59.336532 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:09:59.336618 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:09:59.336638 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:09:59.336720 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:09:59.336975 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 20:09:59.337286 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":392,"threads_started":1}
I20260620 20:09:59.341252 10215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68: No bootstrap required, opened a new log
I20260620 20:09:59.342105 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:09:59.349156 10215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491616eb440643b39a0998f41adf1a68" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32997 } }
I20260620 20:09:59.349558 10215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:09:59.349689 10215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491616eb440643b39a0998f41adf1a68, State: Initialized, Role: FOLLOWER
I20260620 20:09:59.350273 10215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491616eb440643b39a0998f41adf1a68" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32997 } }
I20260620 20:09:59.350592 10215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:09:59.350732 10215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:09:59.350911 10215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:09:59.353456 10215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491616eb440643b39a0998f41adf1a68" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32997 } }
I20260620 20:09:59.354045 10215 leader_election.cc:302] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 491616eb440643b39a0998f41adf1a68; no voters:
I20260620 20:09:59.354804 10215 leader_election.cc:288] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:09:59.354979 10220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:09:59.356079 10220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [term 1 LEADER]: Becoming Leader. State: Replica: 491616eb440643b39a0998f41adf1a68, State: Running, Role: LEADER
I20260620 20:09:59.356623 10220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491616eb440643b39a0998f41adf1a68" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32997 } }
I20260620 20:09:59.356990 10215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:09:59.359965 10221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "491616eb440643b39a0998f41adf1a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491616eb440643b39a0998f41adf1a68" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32997 } } }
I20260620 20:09:59.359956 10222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 491616eb440643b39a0998f41adf1a68. Latest consensus state: current_term: 1 leader_uuid: "491616eb440643b39a0998f41adf1a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491616eb440643b39a0998f41adf1a68" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 32997 } } }
I20260620 20:09:59.360383 10221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:59.360455 10222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68 [sys.catalog]: This master's current role is: LEADER
I20260620 20:09:59.361742 10228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:09:59.367688 10228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:09:59.377023 10228 catalog_manager.cc:1347] Generated new cluster ID: fe0b050707bc46bd80f10b60804703dc
I20260620 20:09:59.377131 10228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:09:59.390693 10228 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:09:59.391863 10228 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:09:59.402045 10228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 491616eb440643b39a0998f41adf1a68: Generated new TSK 0
I20260620 20:09:59.402908 10228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:09:59.425074 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:09:59.417238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.4.169.190:32997)
0620 20:09:59.417480 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:09:59.417487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:09:59.417803 (+ 316us) server_negotiation.cc:407] Connection header received
0620 20:09:59.418078 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:09:59.418082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:09:59.418158 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:09:59.418327 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:09:59.418838 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:59.419562 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:59.420895 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:09:59.421347 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:09:59.421488 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:59.421996 (+ 508us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:09:59.424411 (+ 2415us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:09:59.424750 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55404: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 20:09:59.425200 10218 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55404: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 20:09:59.427321 10244 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:09:59.417108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:32997 (local address 127.0.0.1:55404)
0620 20:09:59.417680 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 20:09:59.417702 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:09:59.417884 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:09:59.418340 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:09:59.418346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:09:59.418357 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:09:59.418672 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:59.418681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:59.419704 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:59.419710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:59.420752 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:09:59.420763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:09:59.421548 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:09:59.421557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:09:59.421707 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:09:59.421744 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:09:59.424667 (+ 2923us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:09:59.426856 (+ 2189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.169.190:32997: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":1831,"thread_start_us":399,"threads_started":1}
I20260620 20:09:59.428365 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 10149
I20260620 20:09:59.429453 10210 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 20:09:59.440119 10245 webserver.cc:533] Webserver: sslize failed:
E20260620 20:09:59.440287 10245 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:09:59.522645 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 10149
2026-06-20T20:09:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (714 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T20:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:09:59Z Disabled control of system clock
I20260620 20:09:59.602921 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45825
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:44381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:45825
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/testchainca.cert with env {}
W20260620 20:09:59.930315 10250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:09:59.930610 10250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:09:59.930699 10250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:09:59.937673 10250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:09:59.937770 10250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:09:59.937809 10250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:09:59.937865 10250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:09:59.937902 10250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:09:59.937963 10250 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:09:59.937995 10250 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 20:09:59.947599 10250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:44381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:45825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:09:59.948828 10250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:09:59.950160 10250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:09:59.962042 10258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:59.962142 10255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:09:59.962121 10256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:09:59.963043 10250 server_base.cc:1034] running on GCE node
I20260620 20:09:59.963799 10250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:09:59.965219 10250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:09:59.966521 10250 hybrid_clock.cc:648] HybridClock initialized: now 1781986199966479 us; error 29 us; skew 500 ppm
I20260620 20:09:59.966847 10250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:09:59.969087 10250 webserver.cc:460] Webserver started at http://127.4.169.190:44437/ using document root <none> and password file <none>
I20260620 20:09:59.969750 10250 fs_manager.cc:359] Metadata directory not provided
I20260620 20:09:59.969835 10250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:09:59.970083 10250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:09:59.972882 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "8c85d3b42b414877a705ab03cd6f43b1"
format_stamp: "Formatted at 2026-06-20 20:09:59 on dist-test-slave-4bwq"
server_key: "80be2477b6b3b85ca08b56232497eb4e"
server_key_iv: "a67d1e334aeac0aa41a1024e4e76665a"
server_key_version: "encryptionkey@0"
I20260620 20:09:59.973621 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "8c85d3b42b414877a705ab03cd6f43b1"
format_stamp: "Formatted at 2026-06-20 20:09:59 on dist-test-slave-4bwq"
server_key: "80be2477b6b3b85ca08b56232497eb4e"
server_key_iv: "a67d1e334aeac0aa41a1024e4e76665a"
server_key_version: "encryptionkey@0"
I20260620 20:09:59.978682 10250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:09:59.981842 10264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:09:59.983170 10250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:09:59.983382 10250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "8c85d3b42b414877a705ab03cd6f43b1"
format_stamp: "Formatted at 2026-06-20 20:09:59 on dist-test-slave-4bwq"
server_key: "80be2477b6b3b85ca08b56232497eb4e"
server_key_iv: "a67d1e334aeac0aa41a1024e4e76665a"
server_key_version: "encryptionkey@0"
I20260620 20:09:59.983583 10250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:09:59.999019 10250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:09:59.999881 10250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:00.000162 10250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:00.019373 10250 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:45825
I20260620 20:10:00.019392 10315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:45825 every 8 connection(s)
I20260620 20:10:00.020725 10250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:10:00.021492 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 10250
I20260620 20:10:00.021725 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:10:00.022527 4774 external_mini_cluster.cc:1428] Setting key aa940e5d9c9992768aa17c090ebdc164
I20260620 20:10:00.025244 10316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:00.037645 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:00.024426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.4.169.190:45825)
0620 20:10:00.025434 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:00.025448 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:00.025498 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:10:00.027782 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:00.027808 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:00.028404 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:00.028822 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:00.029353 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:00.031418 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:00.032662 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:00.033661 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:00.033831 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:00.034901 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:00.034910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:00.034933 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:00.035664 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:00.035760 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:00.035783 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:00.035907 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:00.036100 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 20:10:00.036437 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":298,"threads_started":1}
I20260620 20:10:00.038812 10316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1: Bootstrap starting.
I20260620 20:10:00.041385 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:10:00.042263 10316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:00.043210 10316 log.cc:824] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:00.046005 10316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1: No bootstrap required, opened a new log
I20260620 20:10:00.053026 10316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c85d3b42b414877a705ab03cd6f43b1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45825 } }
I20260620 20:10:00.053429 10316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:00.053563 10316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c85d3b42b414877a705ab03cd6f43b1, State: Initialized, Role: FOLLOWER
I20260620 20:10:00.054117 10316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c85d3b42b414877a705ab03cd6f43b1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45825 } }
I20260620 20:10:00.054345 10316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:10:00.054497 10316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:10:00.054667 10316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:00.056694 10316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c85d3b42b414877a705ab03cd6f43b1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45825 } }
I20260620 20:10:00.057161 10316 leader_election.cc:302] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c85d3b42b414877a705ab03cd6f43b1; no voters:
I20260620 20:10:00.057736 10316 leader_election.cc:288] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:10:00.057914 10321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:00.058642 10321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [term 1 LEADER]: Becoming Leader. State: Replica: 8c85d3b42b414877a705ab03cd6f43b1, State: Running, Role: LEADER
I20260620 20:10:00.059252 10321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c85d3b42b414877a705ab03cd6f43b1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45825 } }
I20260620 20:10:00.059819 10316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:00.062368 10322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c85d3b42b414877a705ab03cd6f43b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c85d3b42b414877a705ab03cd6f43b1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45825 } } }
I20260620 20:10:00.062343 10323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c85d3b42b414877a705ab03cd6f43b1. Latest consensus state: current_term: 1 leader_uuid: "8c85d3b42b414877a705ab03cd6f43b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c85d3b42b414877a705ab03cd6f43b1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45825 } } }
I20260620 20:10:00.062840 10322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:00.062872 10323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:00.068748 10326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:00.073905 10326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:00.082082 10326 catalog_manager.cc:1347] Generated new cluster ID: 66aa68b0f4ce49f391105830e9741991
I20260620 20:10:00.082197 10326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:00.093652 10326 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:10:00.095501 10326 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:00.105346 10326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c85d3b42b414877a705ab03cd6f43b1: Generated new TSK 0
I20260620 20:10:00.106417 10326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:10:00.225090 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:00.128166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.4.169.190:45825)
0620 20:10:00.128376 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:00.128387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:00.128894 (+ 507us) server_negotiation.cc:407] Connection header received
0620 20:10:00.129129 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:00.129135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:00.129234 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:00.129415 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:10:00.129830 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:00.131016 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:00.132368 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:00.133085 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:00.133272 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:00.133735 (+ 463us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:10:00.224363 (+ 90628us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:10:00.224766 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34272: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 20:10:00.225236 10319 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34272: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 20:10:00.225596 10345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:00.128113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45825 (local address 127.0.0.1:34272)
0620 20:10:00.128787 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:00.128803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:00.128924 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:00.129423 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:00.129426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:00.129436 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:10:00.129685 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:00.129693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:00.131160 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:00.131164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:00.132221 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:00.132231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:00.133317 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:00.133322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:00.133437 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:00.133469 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:10:00.224657 (+ 91188us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:10:00.224952 (+ 295us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.169.190:45825: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":378,"threads_started":1}
I20260620 20:10:00.227020 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 10250
I20260620 20:10:00.228145 10311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:00.328639 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 10250
2026-06-20T20:10:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (785 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (409 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (406 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:10:01 dist-test-slave-4bwq krb5kdc[10360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:10:01 dist-test-slave-4bwq krb5kdc[10360](info): set up 2 sockets
Jun 20 20:10:01 dist-test-slave-4bwq krb5kdc[10360](info): commencing operation
krb5kdc: starting...
W20260620 20:10:02.227296 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:10:02 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:10:02.423426 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:35477
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:35477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:02.745011 10373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:02.745370 10373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:02.745532 10373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:02.753108 10373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:02.753265 10373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:02.753369 10373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:02.753472 10373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:02.753549 10373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:02.763888 10373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:35477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:35477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:02.765300 10373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:02.766758 10373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:02.779017 10381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:02.779091 10378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:02.779054 10379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:02.779960 10373 server_base.cc:1034] running on GCE node
I20260620 20:10:02.781023 10373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:02.782612 10373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:02.783855 10373 hybrid_clock.cc:648] HybridClock initialized: now 1781986202783810 us; error 35 us; skew 500 ppm
I20260620 20:10:02.784250 10373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:02 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986202, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:02.787303 10373 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:10:02.788777 10373 webserver.cc:460] Webserver started at http://127.4.169.190:33269/ using document root <none> and password file <none>
I20260620 20:10:02.789444 10373 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:02.789582 10373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:02.789884 10373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:02.792796 10373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "b1317ae0effb46279ce55aae57f081e4"
format_stamp: "Formatted at 2026-06-20 20:10:02 on dist-test-slave-4bwq"
server_key: "d5213dfbd5377ad89bb556b01bc97b61"
server_key_iv: "652831507bc99e83e447247cee8fcdc5"
server_key_version: "encryptionkey@0"
I20260620 20:10:02.793538 10373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "b1317ae0effb46279ce55aae57f081e4"
format_stamp: "Formatted at 2026-06-20 20:10:02 on dist-test-slave-4bwq"
server_key: "d5213dfbd5377ad89bb556b01bc97b61"
server_key_iv: "652831507bc99e83e447247cee8fcdc5"
server_key_version: "encryptionkey@0"
I20260620 20:10:02.798322 10373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 20:10:02.801550 10388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:02.802888 10373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:10:02.803069 10373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "b1317ae0effb46279ce55aae57f081e4"
format_stamp: "Formatted at 2026-06-20 20:10:02 on dist-test-slave-4bwq"
server_key: "d5213dfbd5377ad89bb556b01bc97b61"
server_key_iv: "652831507bc99e83e447247cee8fcdc5"
server_key_version: "encryptionkey@0"
I20260620 20:10:02.803262 10373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:02.819334 10373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:02.823092 10373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:02.823375 10373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:02.841526 10439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:35477 every 8 connection(s)
I20260620 20:10:02.841534 10373 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:35477
I20260620 20:10:02.842978 10373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:10:02.846645 10440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:02.852403 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 10373
I20260620 20:10:02.852636 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:10:02.853241 4774 external_mini_cluster.cc:1428] Setting key ff0b17d1ff1d50f2b19f7c9a31e3514b
I20260620 20:10:02.857584 10440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4: Bootstrap starting.
I20260620 20:10:02.861378 10440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:02.862622 10440 log.cc:824] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:02.865636 10440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4: No bootstrap required, opened a new log
Jun 20 20:10:02 dist-test-slave-4bwq krb5kdc[10360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:02.875298 10440 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1317ae0effb46279ce55aae57f081e4" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35477 } }
I20260620 20:10:02.875733 10440 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:02.875839 10440 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1317ae0effb46279ce55aae57f081e4, State: Initialized, Role: FOLLOWER
I20260620 20:10:02.876535 10440 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1317ae0effb46279ce55aae57f081e4" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35477 } }
I20260620 20:10:02.876796 10440 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:10:02.876942 10440 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:10:02.877141 10440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:02.877964 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:02.855224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.4.169.190:35477)
0620 20:10:02.855993 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:02.856009 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:02.856065 (+ 56us) server_negotiation.cc:407] Connection header received
0620 20:10:02.858442 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:02.858468 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:02.858932 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:02.859397 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:02.860591 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:02.862581 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:02.864201 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:02.865191 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:02.865366 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:02.869157 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:02.869167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:02.869186 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:02.869232 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:02.873371 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:02.874013 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:02.874020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:02.874031 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:02.874247 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:02.874659 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:02.874664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:02.874667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:02.875363 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:02.875626 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:02.876114 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 20:10:02.876542 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":316,"threads_started":1}
I20260620 20:10:02.879876 10440 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1317ae0effb46279ce55aae57f081e4" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35477 } }
I20260620 20:10:02.880362 10440 leader_election.cc:302] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1317ae0effb46279ce55aae57f081e4; no voters:
I20260620 20:10:02.881017 10440 leader_election.cc:288] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:10:02.881139 10445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:02.881834 10445 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [term 1 LEADER]: Becoming Leader. State: Replica: b1317ae0effb46279ce55aae57f081e4, State: Running, Role: LEADER
I20260620 20:10:02.882418 10445 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1317ae0effb46279ce55aae57f081e4" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35477 } }
I20260620 20:10:02.883229 10440 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:02.887902 10446 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1317ae0effb46279ce55aae57f081e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1317ae0effb46279ce55aae57f081e4" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35477 } } }
I20260620 20:10:02.887997 10447 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1317ae0effb46279ce55aae57f081e4. Latest consensus state: current_term: 1 leader_uuid: "b1317ae0effb46279ce55aae57f081e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1317ae0effb46279ce55aae57f081e4" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35477 } } }
I20260620 20:10:02.888288 10446 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:02.888348 10447 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:02.890650 10455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:02.894968 10455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:02.902985 10455 catalog_manager.cc:1347] Generated new cluster ID: b992f9bfefe54febabf30ba5e82cd570
I20260620 20:10:02.903100 10455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:02.922082 10455 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:10:02.923537 10455 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:02.934692 10455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1317ae0effb46279ce55aae57f081e4: Generated new TSK 0
I20260620 20:10:02.935525 10455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:10:02.980063 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35477
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:03.308848 10466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:03.309165 10466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:03.309330 10466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:03.316702 10466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:03.316861 10466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:03.317031 10466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:10:03.326728 10466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:03.328078 10466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:03.329385 10466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:03.342662 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:03.342753 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:03.342847 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:03.343554 10466 server_base.cc:1034] running on GCE node
I20260620 20:10:03.344100 10466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:03.344918 10466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:03.346141 10466 hybrid_clock.cc:648] HybridClock initialized: now 1781986203346079 us; error 32 us; skew 500 ppm
I20260620 20:10:03.346517 10466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:03 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986203, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:03.349582 10466 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:10:03.351101 10466 webserver.cc:460] Webserver started at http://127.4.169.129:40089/ using document root <none> and password file <none>
I20260620 20:10:03.351792 10466 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:03.351935 10466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:03.352254 10466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:03.355146 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "e4d4f18140844e4bb5254febd81a120d"
format_stamp: "Formatted at 2026-06-20 20:10:03 on dist-test-slave-4bwq"
server_key: "e9dbd1511d130df471c736e7f0b44e76"
server_key_iv: "a3fc163c33a1607c61e78aa5b2287d79"
server_key_version: "encryptionkey@0"
I20260620 20:10:03.355924 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "e4d4f18140844e4bb5254febd81a120d"
format_stamp: "Formatted at 2026-06-20 20:10:03 on dist-test-slave-4bwq"
server_key: "e9dbd1511d130df471c736e7f0b44e76"
server_key_iv: "a3fc163c33a1607c61e78aa5b2287d79"
server_key_version: "encryptionkey@0"
I20260620 20:10:03.360934 10466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:10:03.364353 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:03.365625 10466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:03.365815 10466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "e4d4f18140844e4bb5254febd81a120d"
format_stamp: "Formatted at 2026-06-20 20:10:03 on dist-test-slave-4bwq"
server_key: "e9dbd1511d130df471c736e7f0b44e76"
server_key_iv: "a3fc163c33a1607c61e78aa5b2287d79"
server_key_version: "encryptionkey@0"
I20260620 20:10:03.366012 10466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:03.394043 10466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:03.398877 10466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:03.399187 10466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:03.400097 10466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:03.401391 10466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:03.401463 10466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:03.401544 10466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:03.401587 10466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:03.435575 10466 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:44779
I20260620 20:10:03.435627 10593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:44779 every 8 connection(s)
I20260620 20:10:03.436882 10466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:10:03.440181 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 10466
I20260620 20:10:03.440412 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:10:03.441048 4774 external_mini_cluster.cc:1428] Setting key c3f1fb7b373927de5bed1ccdda9e645c
Jun 20 20:10:03 dist-test-slave-4bwq krb5kdc[10360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986203, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:03.457738 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:03.439575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:45005 (local address 127.4.169.190:35477)
0620 20:10:03.440069 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:03.440076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:03.441485 (+ 1409us) server_negotiation.cc:407] Connection header received
0620 20:10:03.443267 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:03.443273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:03.443352 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:03.443475 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:03.444901 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:03.445633 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:03.447244 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:03.447668 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:03.447799 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:03.451756 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:03.451761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:03.451764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:03.451794 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:03.454347 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:03.455332 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:03.455336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:03.455339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:03.455446 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:03.455943 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:03.455946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:03.455948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:03.456213 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:03.456368 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:03.457114 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 20:10:03.457353 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260620 20:10:03.458873 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:03.440182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:35477 (local address 127.4.169.129:45005)
0620 20:10:03.441048 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:03.441093 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:03.443019 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:03.443701 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:03.443709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:03.444189 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:03.444743 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:03.444759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:03.445794 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:03.445798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:03.447080 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:03.447102 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:03.447885 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:03.447894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:03.448171 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:03.448842 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:03.448863 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:03.451532 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:03.454524 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:03.454536 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:03.454556 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:03.455201 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:03.455555 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:03.455562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:03.455569 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:03.455822 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:03.456343 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:03.456351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:03.456729 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 20:10:03.456991 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 20:10:03.457548 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":405,"threads_started":1}
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:10:03.460772 10594 heartbeater.cc:347] Connected to a master server at 127.4.169.190:35477
I20260620 20:10:03.461102 10594 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:03.461907 10594 heartbeater.cc:511] Master 127.4.169.190:35477 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
I20260620 20:10:03.464466 10405 ts_manager.cc:194] Registered new tserver with Master: e4d4f18140844e4bb5254febd81a120d (127.4.169.129:44779)
I20260620 20:10:03.466746 10405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:45005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:10:03.487984 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35477
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:03.813747 10600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:03.814051 10600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:03.814157 10600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:03.821368 10600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:03.821501 10600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:03.821619 10600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:10:03.831705 10600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:03.832968 10600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:03.834225 10600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:03.848390 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:03.848413 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:03.849115 10605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:03.849414 10600 server_base.cc:1034] running on GCE node
I20260620 20:10:03.849979 10600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:03.850975 10600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:03.852188 10600 hybrid_clock.cc:648] HybridClock initialized: now 1781986203852141 us; error 37 us; skew 500 ppm
I20260620 20:10:03.852499 10600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:03 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986203, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:03.860581 10600 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:10:03.862051 10600 webserver.cc:460] Webserver started at http://127.4.169.130:40027/ using document root <none> and password file <none>
I20260620 20:10:03.862805 10600 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:03.862952 10600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:03.863256 10600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:03.866117 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "1570975b503441daa409ba754d5d4d0d"
format_stamp: "Formatted at 2026-06-20 20:10:03 on dist-test-slave-4bwq"
server_key: "998b340cd5614147e2d1a3ba367a3b7a"
server_key_iv: "95d7468e6fa49183585af1bd73f7489f"
server_key_version: "encryptionkey@0"
I20260620 20:10:03.866926 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "1570975b503441daa409ba754d5d4d0d"
format_stamp: "Formatted at 2026-06-20 20:10:03 on dist-test-slave-4bwq"
server_key: "998b340cd5614147e2d1a3ba367a3b7a"
server_key_iv: "95d7468e6fa49183585af1bd73f7489f"
server_key_version: "encryptionkey@0"
I20260620 20:10:03.871847 10600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:10:03.875028 10615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:03.876238 10600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:10:03.876410 10600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "1570975b503441daa409ba754d5d4d0d"
format_stamp: "Formatted at 2026-06-20 20:10:03 on dist-test-slave-4bwq"
server_key: "998b340cd5614147e2d1a3ba367a3b7a"
server_key_iv: "95d7468e6fa49183585af1bd73f7489f"
server_key_version: "encryptionkey@0"
I20260620 20:10:03.876602 10600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:03.911407 10600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:03.915647 10600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:03.915949 10600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:03.916952 10600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:03.918277 10600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:03.918380 10600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:03.918551 10600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:03.918627 10600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:03.952198 10600 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:35715
I20260620 20:10:03.952293 10727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:35715 every 8 connection(s)
I20260620 20:10:03.953436 10600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:10:03.957664 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 10600
I20260620 20:10:03.957868 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:10:03.958289 4774 external_mini_cluster.cc:1428] Setting key b3a11e26ff4b6b6dc8fb89901c501150
Jun 20 20:10:03 dist-test-slave-4bwq krb5kdc[10360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986203, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:03.974114 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:03.955920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:60999 (local address 127.4.169.190:35477)
0620 20:10:03.956098 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:03.956103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:03.957265 (+ 1162us) server_negotiation.cc:407] Connection header received
0620 20:10:03.959124 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:03.959130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:03.959219 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:03.959357 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:03.961084 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:03.962308 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:03.963869 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:03.964626 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:03.964764 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:03.968268 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:03.968274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:03.968277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:03.968309 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:03.971108 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:03.971836 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:03.971844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:03.971849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:03.971981 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:03.972406 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:03.972414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:03.972417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:03.972682 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:03.972839 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:03.973523 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 20:10:03.973755 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:10:03.975319 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:03.956448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:35477 (local address 127.4.169.130:60999)
0620 20:10:03.957119 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:03.957156 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:03.958871 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:03.959609 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:03.959620 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:03.960355 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:03.960912 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:03.960930 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:03.962500 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:03.962504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:03.963725 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:03.963737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:03.964841 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:03.964846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:03.965048 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:03.965746 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:03.965772 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:03.968019 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:03.971268 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:03.971277 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:03.971295 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:03.971716 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:03.972092 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:03.972098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:03.972102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:03.972297 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:03.972808 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:03.972816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:03.973166 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 20:10:03.973440 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 20:10:03.973988 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":277,"threads_started":1}
I20260620 20:10:03.976715 10728 heartbeater.cc:347] Connected to a master server at 127.4.169.190:35477
I20260620 20:10:03.977015 10728 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:10:03.977810 10728 heartbeater.cc:511] Master 127.4.169.190:35477 requested a full tablet report, sending...
I20260620 20:10:03.979292 10405 ts_manager.cc:194] Registered new tserver with Master: 1570975b503441daa409ba754d5d4d0d (127.4.169.130:35715)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
I20260620 20:10:03.980275 10405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:60999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:10:04.003207 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35477
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:04.347342 10733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:04.347627 10733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:04.347724 10733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:04.354976 10733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:04.355077 10733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:04.355194 10733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:10:04.364674 10733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:04.365916 10733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:04.367250 10733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:04.380404 10738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:04.380369 10741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:04.380462 10739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:04.381631 10733 server_base.cc:1034] running on GCE node
I20260620 20:10:04.382241 10733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:04.383275 10733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:04.384490 10733 hybrid_clock.cc:648] HybridClock initialized: now 1781986204384431 us; error 71 us; skew 500 ppm
I20260620 20:10:04.384806 10733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:04.387907 10733 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:10:04.389412 10733 webserver.cc:460] Webserver started at http://127.4.169.131:36433/ using document root <none> and password file <none>
I20260620 20:10:04.390098 10733 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:04.390244 10733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:04.390623 10733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:04.393450 10733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "a4141ea26abd497686afdf89f86e0ae6"
format_stamp: "Formatted at 2026-06-20 20:10:04 on dist-test-slave-4bwq"
server_key: "df3c08d5a1c835ad13704c64e6225ce6"
server_key_iv: "58eaf27cb5a9897d164abbdf74fb3b48"
server_key_version: "encryptionkey@0"
I20260620 20:10:04.394217 10733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "a4141ea26abd497686afdf89f86e0ae6"
format_stamp: "Formatted at 2026-06-20 20:10:04 on dist-test-slave-4bwq"
server_key: "df3c08d5a1c835ad13704c64e6225ce6"
server_key_iv: "58eaf27cb5a9897d164abbdf74fb3b48"
server_key_version: "encryptionkey@0"
I20260620 20:10:04.399240 10733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:10:04.402621 10748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:04.403899 10733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:10:04.404075 10733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "a4141ea26abd497686afdf89f86e0ae6"
format_stamp: "Formatted at 2026-06-20 20:10:04 on dist-test-slave-4bwq"
server_key: "df3c08d5a1c835ad13704c64e6225ce6"
server_key_iv: "58eaf27cb5a9897d164abbdf74fb3b48"
server_key_version: "encryptionkey@0"
I20260620 20:10:04.404278 10733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:04.428633 10733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:04.432848 10733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:04.433156 10733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:04.434144 10733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:04.435633 10733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:04.435736 10733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:04.435889 10733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:04.435961 10733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:04.468736 10733 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:37239
I20260620 20:10:04.468783 10860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:37239 every 8 connection(s)
I20260620 20:10:04.470089 10733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:10:04.470103 10594 heartbeater.cc:503] Master 127.4.169.190:35477 was elected leader, sending a full tablet report...
I20260620 20:10:04.473109 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 10733
I20260620 20:10:04.473315 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:10:04.473778 4774 external_mini_cluster.cc:1428] Setting key f51622ff8be21f87395a664ecc0876cc
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:04.490571 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:04.472938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:42149 (local address 127.4.169.190:35477)
0620 20:10:04.473145 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.473151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.474548 (+ 1397us) server_negotiation.cc:407] Connection header received
0620 20:10:04.476652 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.476657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.476747 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.476859 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.478304 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.479092 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.480693 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.481116 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.481245 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:04.484757 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:04.484763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:04.484768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:04.484807 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:04.487580 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.488299 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.488304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.488308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.488446 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.488857 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.488865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.488871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.489133 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:04.489292 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:04.489976 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 20:10:04.490187 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:10:04.491531 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:04.473475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:35477 (local address 127.4.169.131:42149)
0620 20:10:04.474367 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:04.474440 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:04.476399 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:04.477080 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:04.477088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:04.477570 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:04.478109 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:04.478125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.479244 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.479255 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:04.480545 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:04.480557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.481349 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.481356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:04.481576 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:04.482250 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:04.482272 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:04.484559 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:04.487717 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:04.487725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:04.487741 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:04.488162 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:04.488554 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:04.488560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:04.488563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:04.488748 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:04.489269 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:04.489277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:04.489614 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 20:10:04.489885 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 20:10:04.490456 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":328,"threads_started":1}
I20260620 20:10:04.492830 10861 heartbeater.cc:347] Connected to a master server at 127.4.169.190:35477
I20260620 20:10:04.493134 10861 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:04.493854 10861 heartbeater.cc:511] Master 127.4.169.190:35477 requested a full tablet report, sending...
I20260620 20:10:04.495280 10405 ts_manager.cc:194] Registered new tserver with Master: a4141ea26abd497686afdf89f86e0ae6 (127.4.169.131:37239)
I20260620 20:10:04.496064 10405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:42149
I20260620 20:10:04.500883 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/test-admin.keytab.
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:04.527951 4774 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 20:10:04.533010 10868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:04.530614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.4.169.129:44779)
0620 20:10:04.531275 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.531283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.531310 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:10:04.531368 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.531374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.531591 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.531718 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.532563 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56664: BlockingRecv error: recv got EOF from 127.0.0.1:56664 (error 108)
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":340,"threads_started":1}
W20260620 20:10:04.533164 4774 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:04.534701 4774 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:04.536011 4774 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:10:04.550225 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:04.537602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.4.169.129:44779)
0620 20:10:04.537810 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.537816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.537989 (+ 173us) server_negotiation.cc:407] Connection header received
0620 20:10:04.538166 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.538173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.538234 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.538339 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.539291 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.540210 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.541389 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.541933 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.542072 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:04.545438 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:04.545447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:04.545461 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:04.545490 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:04.548049 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.548630 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.548636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.548638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.548738 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.549038 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.549042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.549043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.549353 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:04.549552 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:04.549796 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 20:10:04.549972 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:10:04.555439 10868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:04.553899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.4.169.129:44779)
0620 20:10:04.554092 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.554098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.554256 (+ 158us) server_negotiation.cc:407] Connection header received
0620 20:10:04.554365 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.554369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.554475 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.554552 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.555198 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56672: BlockingRecv error: recv got EOF from 127.0.0.1:56672 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 20:10:04.555863 4774 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5cc)
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:04.557309 4774 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:04.558583 4774 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:10:04.571401 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:04.559727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.4.169.129:44779)
0620 20:10:04.559918 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.559926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.560170 (+ 244us) server_negotiation.cc:407] Connection header received
0620 20:10:04.560362 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.560371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.560421 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.560514 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.561365 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.562027 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.563325 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.563720 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.563858 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:04.567041 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:04.567046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:04.567049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:04.567097 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:04.569286 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.569952 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.569955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.569958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.570065 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.570382 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.570425 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.570428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.570662 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:04.570858 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:04.571064 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 20:10:04.571198 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 20:10:04.575279 10868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:04.573672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.4.169.129:44779)
0620 20:10:04.573848 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.573853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.574043 (+ 190us) server_negotiation.cc:407] Connection header received
0620 20:10:04.574220 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.574230 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.574296 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.574389 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.575062 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56690: BlockingRecv error: recv got EOF from 127.0.0.1:56690 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 20:10:04.575537 4774 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986152907614-4774-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:04.577039 4774 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:04.578315 4774 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:10:04.593138 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:04.580587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.4.169.129:44779)
0620 20:10:04.580801 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.580809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.581101 (+ 292us) server_negotiation.cc:407] Connection header received
0620 20:10:04.581261 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.581266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.581377 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.581469 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.582435 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.583139 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.584421 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.584867 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.584998 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:04.588439 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:04.588444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:04.588448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:04.588476 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:04.590774 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.591375 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.591387 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.591389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.591502 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.591848 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.591854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.591858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.592114 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:04.592408 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:04.592654 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 20:10:04.592865 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:10:04.594692 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 10466
I20260620 20:10:04.605136 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:04.595338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.4.169.129:44779)
0620 20:10:04.595510 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.595518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.595814 (+ 296us) server_negotiation.cc:407] Connection header received
0620 20:10:04.595926 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.595929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.595978 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.596049 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.596867 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.597495 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.598670 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.599049 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.599139 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:04.600941 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:04.600945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:04.600948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:04.600969 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:04.603140 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.603660 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.603663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.603665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.603766 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.604042 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.604045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.604047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.604267 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:04.604443 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:04.604674 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 20:10:04.604833 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:10:04.605660 10589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:04.691227 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 10466
I20260620 20:10:04.708161 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 10600
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:10:04.723449 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:04.709129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.4.169.130:35715)
0620 20:10:04.709706 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.709713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.709738 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:10:04.709799 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.709805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.710007 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.710142 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.711099 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.712107 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.713226 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.713767 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.713860 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:04.717622 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:04.717634 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:04.717649 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:04.717687 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:04.720675 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.721393 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.721400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.721403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.721538 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.721932 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.721935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.721937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.722313 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:04.722711 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:04.722914 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 20:10:04.723103 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":331,"threads_started":1}
I20260620 20:10:04.724717 10723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:04.806874 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 10600
I20260620 20:10:04.822662 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 10733
Jun 20 20:10:04 dist-test-slave-4bwq krb5kdc[10360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:10:04.837728 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:04.823717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.4.169.131:37239)
0620 20:10:04.824338 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:04.824345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:04.824373 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:10:04.824431 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:04.824437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:04.824644 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:04.824782 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:04.825733 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.826819 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.828010 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:04.828548 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:04.828647 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:04.832613 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:04.832623 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:04.832637 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:04.832664 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:04.835201 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.835814 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.835821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.835823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.835934 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:04.836294 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:04.836299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:04.836303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:04.836615 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:04.836925 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:04.837173 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 20:10:04.837428 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":313,"threads_started":1}
I20260620 20:10:04.838991 10856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:04.928241 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 10733
I20260620 20:10:04.943341 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 10373
I20260620 20:10:04.944499 10435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:05.049247 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 10373
2026-06-20T20:10:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3910 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:10:05 dist-test-slave-4bwq krb5kdc[10888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:10:05 dist-test-slave-4bwq krb5kdc[10888](info): set up 2 sockets
Jun 20 20:10:05 dist-test-slave-4bwq krb5kdc[10888](info): commencing operation
krb5kdc: starting...
W20260620 20:10:06.143319 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:10:06 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:10:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.190.keytab.
Entry for principal oryx/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.190.keytab.
I20260620 20:10:06.348001 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:38587
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.190.keytab
--principal=oryx/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:38587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:06.673485 10901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:06.673774 10901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:06.673866 10901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:06.681053 10901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:06.681178 10901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:06.681226 10901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:06.681277 10901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:06.681320 10901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:06.691584 10901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:38587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.190.keytab
--principal=oryx/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:38587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:06.692840 10901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:06.694070 10901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:06.707341 10906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:06.707392 10909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:06.707315 10907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:06.707827 10901 server_base.cc:1034] running on GCE node
I20260620 20:10:06.708606 10901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:06.709910 10901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:06.711097 10901 hybrid_clock.cc:648] HybridClock initialized: now 1781986206711042 us; error 54 us; skew 500 ppm
I20260620 20:10:06.711395 10901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:06 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:06.714464 10901 init.cc:377] Logged in from keytab as oryx/127.4.169.190@KRBTEST.COM (short username oryx)
I20260620 20:10:06.715857 10901 webserver.cc:460] Webserver started at http://127.4.169.190:42073/ using document root <none> and password file <none>
I20260620 20:10:06.716514 10901 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:06.716642 10901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:06.716948 10901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:06.719907 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "e7c832b7122d493794c7c2731fa72596"
format_stamp: "Formatted at 2026-06-20 20:10:06 on dist-test-slave-4bwq"
server_key: "912629cf991ddb93206d26652decbdde"
server_key_iv: "60b41226e9574db69f0efe614ba176f3"
server_key_version: "encryptionkey@0"
I20260620 20:10:06.720657 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "e7c832b7122d493794c7c2731fa72596"
format_stamp: "Formatted at 2026-06-20 20:10:06 on dist-test-slave-4bwq"
server_key: "912629cf991ddb93206d26652decbdde"
server_key_iv: "60b41226e9574db69f0efe614ba176f3"
server_key_version: "encryptionkey@0"
I20260620 20:10:06.725692 10901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:10:06.728997 10916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:06.730367 10901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:06.730610 10901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "e7c832b7122d493794c7c2731fa72596"
format_stamp: "Formatted at 2026-06-20 20:10:06 on dist-test-slave-4bwq"
server_key: "912629cf991ddb93206d26652decbdde"
server_key_iv: "60b41226e9574db69f0efe614ba176f3"
server_key_version: "encryptionkey@0"
I20260620 20:10:06.730816 10901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:06.744405 10901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:06.748333 10901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:06.748616 10901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:06.766839 10901 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:38587
I20260620 20:10:06.766880 10967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:38587 every 8 connection(s)
I20260620 20:10:06.768164 10901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:10:06.768637 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 10901
I20260620 20:10:06.768898 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:10:06.769671 4774 external_mini_cluster.cc:1428] Setting key bb0c03e5b337f1b90a470c4f07c697f4
I20260620 20:10:06.772455 10968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 20:10:06 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:06.784956 10968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: Bootstrap starting.
I20260620 20:10:06.788410 10968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:06.789397 10968 log.cc:824] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:06.792037 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:06.771744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.4.169.190:38587)
0620 20:10:06.772596 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:06.772612 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:06.772661 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:10:06.774172 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:06.774192 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:06.774623 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:06.774946 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:06.776609 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:06.777910 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:06.779785 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:06.780442 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:06.780571 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:06.785108 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:06.785120 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:06.785144 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:06.785183 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:06.788196 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:06.788921 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:06.788927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:06.788933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:06.789090 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:06.789472 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:06.789477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:06.789481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:06.789980 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:06.790337 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:06.790743 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 20:10:06.791106 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":415,"threads_started":1}
I20260620 20:10:06.792294 10968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: No bootstrap required, opened a new log
I20260620 20:10:06.799403 10968 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } }
I20260620 20:10:06.799804 10968 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:06.799933 10968 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c832b7122d493794c7c2731fa72596, State: Initialized, Role: FOLLOWER
I20260620 20:10:06.800603 10968 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } }
I20260620 20:10:06.800879 10968 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:10:06.801030 10968 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:10:06.801213 10968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:06.803280 10968 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } }
I20260620 20:10:06.803730 10968 leader_election.cc:302] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7c832b7122d493794c7c2731fa72596; no voters:
I20260620 20:10:06.804281 10968 leader_election.cc:288] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:10:06.804402 10973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:06.805099 10973 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 1 LEADER]: Becoming Leader. State: Replica: e7c832b7122d493794c7c2731fa72596, State: Running, Role: LEADER
I20260620 20:10:06.805692 10973 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } }
I20260620 20:10:06.806382 10968 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:06.808480 10975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c832b7122d493794c7c2731fa72596. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } }
I20260620 20:10:06.808969 10975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:06.808848 10974 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } }
I20260620 20:10:06.809113 10974 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:06.811740 10980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:06.816412 10980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:06.825423 10980 catalog_manager.cc:1347] Generated new cluster ID: db37cee9ee884b908d24568ad4881b07
I20260620 20:10:06.825531 10980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:06.835494 10980 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:10:06.836686 10980 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:06.848680 10980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: Generated new TSK 0
I20260620 20:10:06.849524 10980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab.
Entry for principal oryx/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab.
I20260620 20:10:06.909466 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38587
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab
--principal=oryx/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:07.241781 10994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:07.242051 10994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:07.242143 10994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:07.249344 10994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:07.249451 10994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:07.249562 10994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:10:07.259053 10994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab
--principal=oryx/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:07.260183 10994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:07.261492 10994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:07.274685 11002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:07.274613 11000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:07.274849 10999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:07.275336 10994 server_base.cc:1034] running on GCE node
I20260620 20:10:07.275915 10994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:07.276788 10994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:07.278012 10994 hybrid_clock.cc:648] HybridClock initialized: now 1781986207277967 us; error 33 us; skew 500 ppm
I20260620 20:10:07.278317 10994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:07 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986207, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:07.281317 10994 init.cc:377] Logged in from keytab as oryx/127.4.169.129@KRBTEST.COM (short username oryx)
I20260620 20:10:07.282693 10994 webserver.cc:460] Webserver started at http://127.4.169.129:46537/ using document root <none> and password file <none>
I20260620 20:10:07.283324 10994 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:07.283416 10994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:07.283650 10994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:07.286580 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "c3e85269446e48feb97c9899cc44b756"
format_stamp: "Formatted at 2026-06-20 20:10:07 on dist-test-slave-4bwq"
server_key: "8707ec96d54bfadf40092f5d4e73b0d4"
server_key_iv: "992ae38fafd72aab6dd1e5a879d24af8"
server_key_version: "encryptionkey@0"
I20260620 20:10:07.287289 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "c3e85269446e48feb97c9899cc44b756"
format_stamp: "Formatted at 2026-06-20 20:10:07 on dist-test-slave-4bwq"
server_key: "8707ec96d54bfadf40092f5d4e73b0d4"
server_key_iv: "992ae38fafd72aab6dd1e5a879d24af8"
server_key_version: "encryptionkey@0"
I20260620 20:10:07.292275 10994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:10:07.295809 11009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:07.297170 10994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:10:07.297443 10994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "c3e85269446e48feb97c9899cc44b756"
format_stamp: "Formatted at 2026-06-20 20:10:07 on dist-test-slave-4bwq"
server_key: "8707ec96d54bfadf40092f5d4e73b0d4"
server_key_iv: "992ae38fafd72aab6dd1e5a879d24af8"
server_key_version: "encryptionkey@0"
I20260620 20:10:07.297646 10994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:07.310309 10994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:07.314227 10994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:07.314574 10994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:07.315513 10994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:07.316931 10994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:07.317003 10994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:07.317164 10994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:07.317232 10994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:07.351074 10994 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:35319
I20260620 20:10:07.351110 11121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:35319 every 8 connection(s)
I20260620 20:10:07.352420 10994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:10:07.362684 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 10994
I20260620 20:10:07.362936 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:10:07.363606 4774 external_mini_cluster.cc:1428] Setting key ad2dc6bcff61d0f56a23057764599afe
Jun 20 20:10:07 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986207, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:07.372117 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:07.354840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:39811 (local address 127.4.169.190:38587)
0620 20:10:07.355290 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:07.355296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:07.356245 (+ 949us) server_negotiation.cc:407] Connection header received
0620 20:10:07.358009 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:07.358014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:07.358105 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:07.358230 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:07.359771 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:07.360487 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:07.362029 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:07.362680 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:07.362845 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:07.366517 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:07.366522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:07.366525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:07.366555 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:07.369046 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:07.369786 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:07.369794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:07.369797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:07.369918 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:07.370365 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:07.370370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:07.370373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:07.370675 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:07.370833 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:07.371529 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 20:10:07.371752 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260620 20:10:07.373083 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:07.355398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.129:39811)
0620 20:10:07.356073 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:07.356111 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:07.357731 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:07.358501 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:07.358510 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:07.359013 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:07.359583 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:07.359599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:07.360643 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:07.360649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:07.361877 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:07.361888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:07.362943 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:07.362948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:07.363202 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:07.363924 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:07.363945 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:07.366273 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:07.369218 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:07.369227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:07.369244 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:07.369662 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:07.370037 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:07.370042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:07.370046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:07.370243 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:07.370819 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:07.370827 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:07.371170 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 20:10:07.371441 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 20:10:07.371971 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":261,"threads_started":1}
I20260620 20:10:07.374717 11122 heartbeater.cc:347] Connected to a master server at 127.4.169.190:38587
I20260620 20:10:07.375046 11122 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:07.375901 11122 heartbeater.cc:511] Master 127.4.169.190:38587 requested a full tablet report, sending...
I20260620 20:10:07.378163 10933 ts_manager.cc:194] Registered new tserver with Master: c3e85269446e48feb97c9899cc44b756 (127.4.169.129:35319)
I20260620 20:10:07.380363 10933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:39811
WARNING: no policy specified for oryx/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab.
Entry for principal oryx/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab.
I20260620 20:10:07.409829 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38587
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab
--principal=oryx/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:07.742164 11128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:07.742472 11128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:07.742568 11128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:07.749796 11128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:07.749912 11128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:07.750027 11128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:10:07.760097 11128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab
--principal=oryx/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:07.761353 11128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:07.762661 11128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:07.776139 11134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:07.776190 11133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:07.779379 11136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:07.780213 11128 server_base.cc:1034] running on GCE node
I20260620 20:10:07.780851 11128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:07.781831 11128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:07.783070 11128 hybrid_clock.cc:648] HybridClock initialized: now 1781986207783004 us; error 59 us; skew 500 ppm
I20260620 20:10:07.783439 11128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:07 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986207, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:07.786751 11128 init.cc:377] Logged in from keytab as oryx/127.4.169.130@KRBTEST.COM (short username oryx)
I20260620 20:10:07.788295 11128 webserver.cc:460] Webserver started at http://127.4.169.130:37863/ using document root <none> and password file <none>
I20260620 20:10:07.788982 11128 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:07.789119 11128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:07.789427 11128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:07.792348 11128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "f359aa5fee0d4e49949dc64a62e96dbb"
format_stamp: "Formatted at 2026-06-20 20:10:07 on dist-test-slave-4bwq"
server_key: "0d3ae442c022227b1b5ce53351b1bc4e"
server_key_iv: "8c81a5b51fe858be179d3eb6954979c6"
server_key_version: "encryptionkey@0"
I20260620 20:10:07.793097 11128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "f359aa5fee0d4e49949dc64a62e96dbb"
format_stamp: "Formatted at 2026-06-20 20:10:07 on dist-test-slave-4bwq"
server_key: "0d3ae442c022227b1b5ce53351b1bc4e"
server_key_iv: "8c81a5b51fe858be179d3eb6954979c6"
server_key_version: "encryptionkey@0"
I20260620 20:10:07.798372 11128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:10:07.801940 11143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:07.803350 11128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:10:07.803503 11128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "f359aa5fee0d4e49949dc64a62e96dbb"
format_stamp: "Formatted at 2026-06-20 20:10:07 on dist-test-slave-4bwq"
server_key: "0d3ae442c022227b1b5ce53351b1bc4e"
server_key_iv: "8c81a5b51fe858be179d3eb6954979c6"
server_key_version: "encryptionkey@0"
I20260620 20:10:07.803701 11128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:07.837759 11128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:07.842173 11128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:07.842478 11128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:07.843451 11128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:07.844825 11128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:07.844897 11128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:07.845075 11128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:07.845127 11128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:07.878311 11128 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:41699
I20260620 20:10:07.878440 11255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:41699 every 8 connection(s)
I20260620 20:10:07.879563 11128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:10:07.880667 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 11128
I20260620 20:10:07.880877 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:10:07.881431 4774 external_mini_cluster.cc:1428] Setting key 2710ce68ea0808513176cf197b9b9664
Jun 20 20:10:07 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986207, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:07.900154 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:07.882342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:40867 (local address 127.4.169.190:38587)
0620 20:10:07.882827 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:07.882833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:07.883873 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 20:10:07.885650 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:07.885658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:07.885742 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:07.885856 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:07.887427 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:07.888633 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:07.890557 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:07.890978 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:07.891113 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:07.894769 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:07.894774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:07.894778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:07.894807 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:07.897234 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:07.897928 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:07.897934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:07.897944 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:07.898060 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:07.898529 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:07.898532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:07.898534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:07.898772 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:07.898943 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:07.899560 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 20:10:07.899764 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":226,"threads_started":1}
WARNING: no policy specified for oryx/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:10:07.901137 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:07.882927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.130:40867)
0620 20:10:07.883711 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:07.883761 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:07.885392 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:07.886120 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:07.886137 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:07.886697 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:07.887260 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:07.887277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:07.888793 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:07.888801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:07.890360 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:07.890371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:07.891246 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:07.891255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:07.891479 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:07.892136 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:07.892162 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:07.894532 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:07.897363 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:07.897374 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:07.897390 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:07.897805 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:07.898182 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:07.898187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:07.898191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:07.898368 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:07.898904 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:07.898913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:07.899229 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 20:10:07.899473 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 20:10:07.899988 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":354,"threads_started":1}
I20260620 20:10:07.902709 11256 heartbeater.cc:347] Connected to a master server at 127.4.169.190:38587
I20260620 20:10:07.903016 11256 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.131@KRBTEST.COM" created.
I20260620 20:10:07.903847 11256 heartbeater.cc:511] Master 127.4.169.190:38587 requested a full tablet report, sending...
I20260620 20:10:07.905318 10933 ts_manager.cc:194] Registered new tserver with Master: f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699)
I20260620 20:10:07.906090 10933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:40867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab.
Entry for principal oryx/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab.
I20260620 20:10:07.927531 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38587
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab
--principal=oryx/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:08.251550 11262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:08.251863 11262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:08.252029 11262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:08.259560 11262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:08.259686 11262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:08.259912 11262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:10:08.270292 11262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab
--principal=oryx/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:08.271669 11262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:08.272938 11262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:08.286249 11270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:08.286311 11268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:08.286311 11267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:08.287272 11262 server_base.cc:1034] running on GCE node
I20260620 20:10:08.287835 11262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:08.288825 11262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:08.290076 11262 hybrid_clock.cc:648] HybridClock initialized: now 1781986208290025 us; error 32 us; skew 500 ppm
I20260620 20:10:08.290506 11262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:08 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986208, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:08.293565 11262 init.cc:377] Logged in from keytab as oryx/127.4.169.131@KRBTEST.COM (short username oryx)
I20260620 20:10:08.295069 11262 webserver.cc:460] Webserver started at http://127.4.169.131:43147/ using document root <none> and password file <none>
I20260620 20:10:08.295800 11262 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:08.295897 11262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:08.296135 11262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:08.299007 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "23d7e639f3ee49efab35348707f2a64e"
format_stamp: "Formatted at 2026-06-20 20:10:08 on dist-test-slave-4bwq"
server_key: "69b91401d92913bd01c273b1077b1ee3"
server_key_iv: "6c9e5ff6d7f9040527be4ade21ccb939"
server_key_version: "encryptionkey@0"
I20260620 20:10:08.299695 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "23d7e639f3ee49efab35348707f2a64e"
format_stamp: "Formatted at 2026-06-20 20:10:08 on dist-test-slave-4bwq"
server_key: "69b91401d92913bd01c273b1077b1ee3"
server_key_iv: "6c9e5ff6d7f9040527be4ade21ccb939"
server_key_version: "encryptionkey@0"
I20260620 20:10:08.304836 11262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 20:10:08.308637 11277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:08.310024 11262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:08.310233 11262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "23d7e639f3ee49efab35348707f2a64e"
format_stamp: "Formatted at 2026-06-20 20:10:08 on dist-test-slave-4bwq"
server_key: "69b91401d92913bd01c273b1077b1ee3"
server_key_iv: "6c9e5ff6d7f9040527be4ade21ccb939"
server_key_version: "encryptionkey@0"
I20260620 20:10:08.310494 11262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:08.339583 11262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:08.343672 11262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:08.343962 11262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:08.344897 11262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:08.346273 11262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:08.346345 11262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:08.346529 11262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:08.346580 11262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:08.379457 11262 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:40573
I20260620 20:10:08.379503 11389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:40573 every 8 connection(s)
I20260620 20:10:08.380733 11262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:10:08.383988 11122 heartbeater.cc:503] Master 127.4.169.190:38587 was elected leader, sending a full tablet report...
I20260620 20:10:08.387040 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 11262
I20260620 20:10:08.387205 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:10:08.387619 4774 external_mini_cluster.cc:1428] Setting key 43933e2bf30339972be8599b2d5134c9
Jun 20 20:10:08 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986208, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:08.401970 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.383460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:40021 (local address 127.4.169.190:38587)
0620 20:10:08.383680 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:08.383687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:08.385021 (+ 1334us) server_negotiation.cc:407] Connection header received
0620 20:10:08.388274 (+ 3253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:08.388279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:08.388373 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:08.388488 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:08.389965 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.390767 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.392269 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.392646 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.392793 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.396164 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:08.396169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:08.396172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:08.396201 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:08.398867 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.399576 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.399583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.399587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.399705 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.400129 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.400137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.400141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.400402 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:08.400573 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:08.401296 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 20:10:08.401546 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:10:08.403345 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.384019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:40021)
0620 20:10:08.384857 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:08.384910 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:08.388000 (+ 3090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:08.388729 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:08.388737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:08.389233 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:08.389802 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.389819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.390918 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.390928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.392126 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.392137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.392861 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.392868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.393099 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.393778 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:08.393801 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:08.395958 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:08.399003 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.399011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.399028 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.399455 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.399827 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.399832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.399836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.400019 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.400526 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:08.400534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:08.400917 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 20:10:08.401183 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 20:10:08.402064 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":342,"threads_started":1}
I20260620 20:10:08.404570 11390 heartbeater.cc:347] Connected to a master server at 127.4.169.190:38587
I20260620 20:10:08.404863 11390 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:08.405532 11390 heartbeater.cc:511] Master 127.4.169.190:38587 requested a full tablet report, sending...
I20260620 20:10:08.406996 10933 ts_manager.cc:194] Registered new tserver with Master: 23d7e639f3ee49efab35348707f2a64e (127.4.169.131:40573)
I20260620 20:10:08.407917 10933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:40021
I20260620 20:10:08.415509 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.169.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.189.keytab.
Entry for principal oryx/127.4.169.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.189.keytab.
I20260620 20:10:08.461933 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.189:36339
--webserver_interface=127.4.169.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.189.keytab
--principal=oryx/127.4.169.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:38587,127.4.169.189:36339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:08.784790 11395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:08.785059 11395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:08.785156 11395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:08.792446 11395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:08.792546 11395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:08.792590 11395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:08.792646 11395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:08.792690 11395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:08.802460 11395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:38587,127.4.169.189:36339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.189.keytab
--principal=oryx/127.4.169.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.189:36339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.169.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
W20260620 20:10:08.802763 11395 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.169.190:38587,127.4.169.189:36339'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 20:10:08.803771 11395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:08.805044 11395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:08.817535 11400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:08.817473 11403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:08.817431 11401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:08.818208 11395 server_base.cc:1034] running on GCE node
I20260620 20:10:08.819123 11395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:08.820590 11395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:08.821811 11395 hybrid_clock.cc:648] HybridClock initialized: now 1781986208821741 us; error 45 us; skew 500 ppm
I20260620 20:10:08.822147 11395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:08 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986208, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:08.825434 11395 init.cc:377] Logged in from keytab as oryx/127.4.169.189@KRBTEST.COM (short username oryx)
I20260620 20:10:08.827018 11395 webserver.cc:460] Webserver started at http://127.4.169.189:44181/ using document root <none> and password file <none>
I20260620 20:10:08.827776 11395 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:08.827903 11395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:08.828219 11395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:08.831264 11395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data/instance:
uuid: "dc5844a44fc04b1eb9479d71b25ace63"
format_stamp: "Formatted at 2026-06-20 20:10:08 on dist-test-slave-4bwq"
server_key: "c018b9d15668d2751847a2c611d0ae78"
server_key_iv: "e3a5fa00fa0c54e18c4c6a8f7cc49569"
server_key_version: "encryptionkey@0"
I20260620 20:10:08.832015 11395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal/instance:
uuid: "dc5844a44fc04b1eb9479d71b25ace63"
format_stamp: "Formatted at 2026-06-20 20:10:08 on dist-test-slave-4bwq"
server_key: "c018b9d15668d2751847a2c611d0ae78"
server_key_iv: "e3a5fa00fa0c54e18c4c6a8f7cc49569"
server_key_version: "encryptionkey@0"
I20260620 20:10:08.837234 11395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:10:08.840531 11410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:08.841910 11395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:08.842087 11395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal
uuid: "dc5844a44fc04b1eb9479d71b25ace63"
format_stamp: "Formatted at 2026-06-20 20:10:08 on dist-test-slave-4bwq"
server_key: "c018b9d15668d2751847a2c611d0ae78"
server_key_iv: "e3a5fa00fa0c54e18c4c6a8f7cc49569"
server_key_version: "encryptionkey@0"
I20260620 20:10:08.842296 11395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:08.856117 11395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:08.864382 11395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:08.864673 11395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:08.883721 11395 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.189:36339
I20260620 20:10:08.883749 11461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.189:36339 every 8 connection(s)
I20260620 20:10:08.885108 11395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
I20260620 20:10:08.889169 11462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:08.891532 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 11395
I20260620 20:10:08.897755 11462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:10:08 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.189@KRBTEST.COM
I20260620 20:10:08.909076 11256 heartbeater.cc:503] Master 127.4.169.190:38587 was elected leader, sending a full tablet report...
Jun 20 20:10:08 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986208, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.189@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:08.917122 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.900619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.4.169.190:38587)
0620 20:10:08.900811 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:08.900817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:08.901461 (+ 644us) server_negotiation.cc:407] Connection header received
0620 20:10:08.901633 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:08.901639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:08.901728 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:08.901851 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:08.903041 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.904062 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.906133 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.906667 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.906773 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.910698 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:08.910705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:08.910710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:08.910753 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:08.914052 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.914910 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.914914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.914917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.915100 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.915607 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.915612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.915614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.916038 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:08.916230 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:08.916705 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 20:10:08.916879 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:10:08.917920 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.900540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:42342)
0620 20:10:08.901318 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:08.901359 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:08.901511 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:08.901918 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:08.901923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:08.902630 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:08.902901 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.902910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.904200 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.904205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.905911 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.905928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.906919 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.906928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.907175 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.908040 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:08.908063 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:08.910505 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:08.914192 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.914233 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.914239 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.914802 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.915215 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.915220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.915223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.915361 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.916201 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:08.916208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:08.916375 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 20:10:08.916570 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 20:10:08.916890 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":382,"threads_started":1}
I20260620 20:10:08.917943 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.893673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.4.169.189:36339)
0620 20:10:08.894543 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:08.894563 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:08.894613 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:10:08.896941 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:08.896971 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:08.897499 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:08.897940 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:08.899334 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.901175 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.902793 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.903728 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.904302 (+ 574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.909104 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:08.909119 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:08.909140 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:08.909183 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:08.913034 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.913811 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.913822 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.913833 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.914032 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.914478 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.914483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.914486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.915213 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:08.915522 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:08.915989 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 20:10:08.916455 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":372,"threads_started":1}
I20260620 20:10:08.920151 11462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:10:08 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986208, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.189@KRBTEST.COM for oryx/127.4.169.189@KRBTEST.COM
I20260620 20:10:08.931950 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.921079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.0.0.1:58554)
0620 20:10:08.921295 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:08.921316 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:08.921415 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:08.921652 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:08.921655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:08.921954 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:08.922080 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.922087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.923084 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.923088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.923952 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.923963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.924572 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.924576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.924723 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.925242 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:08.925258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:08.927256 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:08.929997 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.930000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.930003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.930349 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.930724 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.930727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.930729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.930849 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.931290 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:08.931293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:08.931411 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 20:10:08.931517 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:10:08.931711 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 20:10:08.932072 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.921299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.4.169.189:36339)
0620 20:10:08.921436 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:08.921440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:08.921456 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:10:08.921493 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:08.921496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:08.921546 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:08.921621 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:08.922188 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.922953 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.924066 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.924458 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.924549 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.927417 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:08.927422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:08.927425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:08.927452 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:08.929884 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.930515 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.930519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.930522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.930634 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.930935 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.930938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.930940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.931199 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:08.931309 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:08.931596 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 20:10:08.931811 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":44}
I20260620 20:10:08.936859 11462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Bootstrap starting.
I20260620 20:10:08.940080 11462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:08.941187 11462 log.cc:824] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:08.944164 11462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: No bootstrap required, opened a new log
I20260620 20:10:08.951457 11462 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } }
I20260620 20:10:08.951802 11462 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:08.951905 11462 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5844a44fc04b1eb9479d71b25ace63, State: Initialized, Role: FOLLOWER
I20260620 20:10:08.952548 11462 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } }
I20260620 20:10:08.953395 11468 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } } }
I20260620 20:10:08.953727 11468 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:10:08.954540 11462 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:10:08.963477 11479 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:10:08.963651 11479 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:10:08.965988 11395 master_runner.cc:428] Detected that this master dc5844a44fc04b1eb9479d71b25ace63 is joining an existing cluster
I20260620 20:10:08.966091 11395 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.169.189:36339
I20260620 20:10:08.990077 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.970314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:42348)
0620 20:10:08.971149 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:08.971170 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:08.971374 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:08.972551 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:08.972556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:08.973222 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:08.973582 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.973594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.975065 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.975069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.977088 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.977102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.978334 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.978339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.978550 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.979299 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:08.979330 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:08.980734 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:08.986482 (+ 5748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.986488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.986494 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.987112 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.987529 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.987536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.987542 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.987722 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.988643 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:08.988651 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:08.988907 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 20:10:08.989057 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 20:10:08.989437 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":164,"spinlock_wait_cycles":1408,"thread_start_us":455,"threads_started":1}
I20260620 20:10:08.990324 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.970322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.4.169.189:36339)
0620 20:10:08.970544 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:08.970552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:08.971818 (+ 1266us) server_negotiation.cc:407] Connection header received
0620 20:10:08.972032 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:08.972036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:08.972105 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:08.972179 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:08.973685 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.974843 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.977232 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.977924 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.978018 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.980924 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:08.980931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:08.980935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:08.980967 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:08.986978 (+ 6011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.987853 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.987863 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.987867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.987989 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.988551 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.988555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.988559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.989002 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:08.989157 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:08.989657 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 20:10:08.989887 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:10:08.991168 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.971043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.0.0.1:58570)
0620 20:10:08.971712 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:08.971734 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:08.971902 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:08.972367 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:08.972372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:08.973108 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:08.973531 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.973547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.975063 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.975068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.977088 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:08.977102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.978313 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.978321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:08.978533 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.979411 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:08.979433 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:08.980734 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:08.987141 (+ 6407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.987147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.987152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.987721 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.988265 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:08.988269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:08.988273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:08.988432 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:08.989245 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:08.989249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:08.989405 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 20:10:08.990414 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0620 20:10:08.990736 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":185,"thread_start_us":346,"threads_started":1}
I20260620 20:10:08.991791 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.970293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.4.169.190:38587)
0620 20:10:08.970505 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:08.970512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:08.971265 (+ 753us) server_negotiation.cc:407] Connection header received
0620 20:10:08.971816 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:08.971823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:08.971904 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:08.971979 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:08.973749 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.974875 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.977247 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:08.977924 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:08.978018 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:08.980939 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:08.980945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:08.980950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:08.980988 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:08.984920 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.987210 (+ 2290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.987218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.987224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.987397 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:08.987866 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:08.987873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:08.987879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:08.988437 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:08.988583 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:08.990612 (+ 2029us) server_negotiation.cc:299] Negotiation successful
0620 20:10:08.990870 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for oryx/127.4.169.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.188@KRBTEST.COM" created.
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.190@KRBTEST.COM for oryx/127.4.169.189@KRBTEST.COM
I20260620 20:10:09.017963 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.997758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.0.0.1:58582)
0620 20:10:08.998991 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:08.999045 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:08.999286 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:08.999940 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:08.999948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.000825 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.001118 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.001130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.002357 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.002364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.004219 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.004239 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.005625 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.005633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.005898 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.006728 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.006783 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.009848 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.014668 (+ 4820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.014681 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.014689 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.015308 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.015755 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.015761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.015765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.015927 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.016636 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.016644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.016887 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.017101 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.017496 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":781,"threads_started":1}
I20260620 20:10:09.017997 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:08.997815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.4.169.189:36339)
0620 20:10:08.997997 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:08.998003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:08.999134 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 20:10:08.999511 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:08.999517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:08.999602 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:08.999718 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.001334 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.002151 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.004387 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.005227 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.005340 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.010066 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.010074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.010080 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.010126 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.014488 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.015457 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.015465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.015471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.015636 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.016067 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.016075 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.016082 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.016493 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.016644 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.017238 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.017515 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:10:09.020727 10933 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.169.189:36339: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63"
member_type: NON_VOTER
last_known_addr {
host: "127.4.169.189"
port: 36339
}
attrs {
promote: true
}
}
dest_uuid: "e7c832b7122d493794c7c2731fa72596"
cas_config_opid_index: -1
I20260620 20:10:09.021956 10933 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } }
I20260620 20:10:09.026163 11437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:09.025626 11491 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dc5844a44fc04b1eb9479d71b25ace63 (127.4.169.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } } }
I20260620 20:10:09.030313 11437 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Refusing update from remote peer e7c832b7122d493794c7c2731fa72596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 20:10:09.031806 11491 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.169.189:36339
I20260620 20:10:09.031870 11493 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:09.032584 11395 master.cc:537] Master@127.4.169.189:36339 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.188.keytab.
Entry for principal oryx/127.4.169.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.188.keytab.
I20260620 20:10:09.040210 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.188:40759
--webserver_interface=127.4.169.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.188.keytab
--principal=oryx/127.4.169.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:38587,127.4.169.189:36339,127.4.169.188:40759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:10:09.040679 11492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } } }
I20260620 20:10:09.041028 11492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
W20260620 20:10:09.075384 10920 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 20:10:09.083628 11395 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:09.084095 11395 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:09.084312 11395 tablet_replica.cc:330] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: stopping tablet replica
W20260620 20:10:09.084323 10920 consensus_peers.cc:487] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 -> Peer dc5844a44fc04b1eb9479d71b25ace63 (127.4.169.189:36339): Couldn't send request to peer dc5844a44fc04b1eb9479d71b25ace63. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 20:10:09.090574 11395 master.cc:552] Master@127.4.169.189:36339 shutdown complete.
I20260620 20:10:09.090682 11395 master_runner.cc:305] Clearing existing system tablet
I20260620 20:10:09.091058 11395 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:09.091187 11395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:09.093436 11395 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:10:09.095017 11499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:09.095381 11395 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:09.095676 11395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:10:09.095764 11395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal
uuid: "dc5844a44fc04b1eb9479d71b25ace63"
format_stamp: "Formatted at 2026-06-20 20:10:08 on dist-test-slave-4bwq"
server_key: "c018b9d15668d2751847a2c611d0ae78"
server_key_iv: "e3a5fa00fa0c54e18c4c6a8f7cc49569"
server_key_version: "encryptionkey@0"
I20260620 20:10:09.099299 11395 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:09.104447 11395 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:10:09.104617 11395 log.cc:1196] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 20:10:09.105037 11395 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Deleting consensus metadata
I20260620 20:10:09.105299 11395 master_runner.cc:315] Copying system tablet from 127.4.169.190:38587
I20260620 20:10:09.119499 11395 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: tablet copy: Beginning tablet copy session from remote peer at address 127.4.169.190:38587
I20260620 20:10:09.134145 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.120380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.4.169.190:38587)
0620 20:10:09.120574 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.120583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.122962 (+ 2379us) server_negotiation.cc:407] Connection header received
0620 20:10:09.123195 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.123201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.123279 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.123383 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.124206 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.124951 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.126200 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.126689 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.126831 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.128691 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.128696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.128699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.128729 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.131396 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.132114 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.132122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.132128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.132258 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.132670 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.132675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.132678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.132943 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.133030 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.133512 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.133791 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:10:09.134171 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.120380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:42356)
0620 20:10:09.122830 (+ 2450us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.122852 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.123046 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.123409 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.123414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.123828 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.124047 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.124055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.125094 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.125101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.126045 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.126056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.126835 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.126841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.127018 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.127688 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.127705 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.128484 (+ 779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.131533 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.131538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.131546 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.131998 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.132364 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.132370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.132374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.132518 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.133113 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.133117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.133255 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.133398 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.133712 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2320,"thread_start_us":2191,"threads_started":1}
I20260620 20:10:09.135470 10953 tablet_copy_service.cc:140] P e7c832b7122d493794c7c2731fa72596: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dc5844a44fc04b1eb9479d71b25ace63 ({username='oryx', principal='oryx/127.4.169.189@KRBTEST.COM'} at 127.0.0.1:42356)
I20260620 20:10:09.135803 10953 tablet_copy_service.cc:161] P e7c832b7122d493794c7c2731fa72596: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dc5844a44fc04b1eb9479d71b25ace63 at {username='oryx', principal='oryx/127.4.169.189@KRBTEST.COM'} at 127.0.0.1:42356: session id = dc5844a44fc04b1eb9479d71b25ace63-00000000000000000000000000000000
I20260620 20:10:09.138809 10953 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:10:09.140830 11395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:10:09.145326 11395 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: tablet copy: Starting download of 0 data blocks...
I20260620 20:10:09.145610 11395 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: tablet copy: Starting download of 1 WAL segments...
I20260620 20:10:09.147948 11395 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:10:09.150979 10953 tablet_copy_service.cc:342] P e7c832b7122d493794c7c2731fa72596: Request end of tablet copy session dc5844a44fc04b1eb9479d71b25ace63-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.169.189@KRBTEST.COM'} at 127.0.0.1:42356
I20260620 20:10:09.151161 10953 tablet_copy_service.cc:433] P e7c832b7122d493794c7c2731fa72596: ending tablet copy session dc5844a44fc04b1eb9479d71b25ace63-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dc5844a44fc04b1eb9479d71b25ace63
I20260620 20:10:09.161042 11395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:09.163851 11506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:09.167953 11507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:09.179469 11509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:09.179632 11395 server_base.cc:1034] running on GCE node
I20260620 20:10:09.180137 11395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:09.180806 11395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:09.182060 11395 hybrid_clock.cc:648] HybridClock initialized: now 1781986209181987 us; error 42 us; skew 500 ppm
I20260620 20:10:09.182323 11395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986209, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:09.184337 11395 init.cc:377] Logged in from keytab as oryx/127.4.169.189@KRBTEST.COM (short username oryx)
I20260620 20:10:09.185644 11395 webserver.cc:460] Webserver started at http://127.4.169.189:41203/ using document root <none> and password file <none>
I20260620 20:10:09.185971 11395 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:09.186064 11395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:09.188764 11395 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:10:09.190716 11516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:09.191104 11395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 20:10:09.191294 11395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal
uuid: "dc5844a44fc04b1eb9479d71b25ace63"
format_stamp: "Formatted at 2026-06-20 20:10:08 on dist-test-slave-4bwq"
server_key: "c018b9d15668d2751847a2c611d0ae78"
server_key_iv: "e3a5fa00fa0c54e18c4c6a8f7cc49569"
server_key_version: "encryptionkey@0"
I20260620 20:10:09.191545 11395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:09.206992 11395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:09.211556 11395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:09.241905 11395 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.189:36339
I20260620 20:10:09.241925 11567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.189:36339 every 8 connection(s)
I20260620 20:10:09.242946 11395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
I20260620 20:10:09.249969 11568 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:10:09.252681 11568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Bootstrap starting.
I20260620 20:10:09.265689 11568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:10:09.266258 11568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Bootstrap complete.
I20260620 20:10:09.267292 11568 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } }
I20260620 20:10:09.267599 11568 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc5844a44fc04b1eb9479d71b25ace63, State: Initialized, Role: LEARNER
I20260620 20:10:09.267831 11568 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } }
I20260620 20:10:09.268633 11571 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } } }
I20260620 20:10:09.268944 11568 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:09.269011 11571 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:10:09.278339 11582 catalog_manager.cc:1259] Loaded cluster ID: db37cee9ee884b908d24568ad4881b07
I20260620 20:10:09.278558 11582 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: loading cluster ID for follower catalog manager: success
I20260620 20:10:09.280941 11582 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: acquiring CA information for follower catalog manager: success
I20260620 20:10:09.282842 11582 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:10:09.410379 11390 heartbeater.cc:503] Master 127.4.169.190:38587 was elected leader, sending a full tablet report...
W20260620 20:10:09.476033 11496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:09.476394 11496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:09.476544 11496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:09.484467 11496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:09.484615 11496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:09.484719 11496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:09.484826 11496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:09.484901 11496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:09.495285 11496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:38587,127.4.169.189:36339,127.4.169.188:40759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.188.keytab
--principal=oryx/127.4.169.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.188:40759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.169.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:09.496675 11496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:09.497979 11496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:09.510378 11587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:09.510443 11590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:09.510447 11588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:09.512279 11496 server_base.cc:1034] running on GCE node
I20260620 20:10:09.513073 11496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:09.514693 11496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:09.516002 11496 hybrid_clock.cc:648] HybridClock initialized: now 1781986209515935 us; error 44 us; skew 500 ppm
I20260620 20:10:09.516398 11496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986209, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:09.519786 11496 init.cc:377] Logged in from keytab as oryx/127.4.169.188@KRBTEST.COM (short username oryx)
I20260620 20:10:09.521615 11496 webserver.cc:460] Webserver started at http://127.4.169.188:44249/ using document root <none> and password file <none>
I20260620 20:10:09.522284 11496 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:09.522429 11496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:09.522722 11496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:09.525723 11496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data/instance:
uuid: "f55538c2647943c1b75637fdb733692b"
format_stamp: "Formatted at 2026-06-20 20:10:09 on dist-test-slave-4bwq"
server_key: "0bb77102813ee4c8a93723262ba4e0e5"
server_key_iv: "8b7b15f98b484b803cb656eaffb2f0c2"
server_key_version: "encryptionkey@0"
I20260620 20:10:09.526525 11496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal/instance:
uuid: "f55538c2647943c1b75637fdb733692b"
format_stamp: "Formatted at 2026-06-20 20:10:09 on dist-test-slave-4bwq"
server_key: "0bb77102813ee4c8a93723262ba4e0e5"
server_key_iv: "8b7b15f98b484b803cb656eaffb2f0c2"
server_key_version: "encryptionkey@0"
I20260620 20:10:09.531795 11496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:10:09.535634 11597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:09.537331 11496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 20:10:09.537546 11496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal
uuid: "f55538c2647943c1b75637fdb733692b"
format_stamp: "Formatted at 2026-06-20 20:10:09 on dist-test-slave-4bwq"
server_key: "0bb77102813ee4c8a93723262ba4e0e5"
server_key_iv: "8b7b15f98b484b803cb656eaffb2f0c2"
server_key_version: "encryptionkey@0"
I20260620 20:10:09.537776 11496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:09.564419 11496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:09.569196 11496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:09.569489 11496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:09.589484 11496 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.188:40759
I20260620 20:10:09.589536 11648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.188:40759 every 8 connection(s)
I20260620 20:10:09.591022 11496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
I20260620 20:10:09.595041 11649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:09.599169 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 11496
I20260620 20:10:09.603336 11649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.188@KRBTEST.COM
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986209, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.188@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:09.625149 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.608003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.4.169.190:38587)
0620 20:10:09.608211 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.608219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.608403 (+ 184us) server_negotiation.cc:407] Connection header received
0620 20:10:09.608620 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.608628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.608718 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.608827 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.610766 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.611732 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.613729 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.614226 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.614327 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.618300 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.618310 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.618317 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.618366 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.621628 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.622527 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.622533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.622536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.622688 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.623174 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.623179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.623182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.623537 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.623687 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.624388 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.624616 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 20:10:09.626098 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.601540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.4.169.188:40759)
0620 20:10:09.602458 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.602476 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.602528 (+ 52us) server_negotiation.cc:407] Connection header received
0620 20:10:09.604404 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.604439 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.605011 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.605478 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.607001 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.608991 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.611141 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.612117 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.612276 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.617792 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.617801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.617814 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.617849 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.621446 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.622545 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.622555 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.622558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.622763 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.623174 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.623178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.623182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.623851 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.624132 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.624508 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.624939 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":258,"server-negotiator.queue_time_us":653,"thread_start_us":412,"threads_started":1}
I20260620 20:10:09.626200 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.607349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:42368)
0620 20:10:09.608268 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.608315 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.608487 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.608888 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.608895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.610253 (+ 1358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.610598 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.610611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.611887 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.611891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.613561 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.613575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.614497 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.614505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.614753 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.615532 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.615565 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.618091 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.621797 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.621809 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.621822 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.622409 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.622803 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.622807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.622809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.623041 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.623655 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.623661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.623894 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.624241 (+ 347us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.624797 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":23,"spinlock_wait_cycles":16256,"thread_start_us":479,"threads_started":1}
I20260620 20:10:09.627895 11649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986209, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.188@KRBTEST.COM for oryx/127.4.169.189@KRBTEST.COM
I20260620 20:10:09.642302 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.635858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.4.169.189:36339)
0620 20:10:09.636468 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.636473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.636497 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:10:09.636540 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.636543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.636602 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.636727 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:09.637195 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.638254 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.639870 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.641214 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.641377 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.641716 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.641823 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.641949 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":351,"threads_started":1}
I20260620 20:10:09.642483 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.635700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.0.0.1:58600)
0620 20:10:09.636171 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.636202 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.636355 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.636777 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.636782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.636814 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:09.637030 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.637041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.638438 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.638449 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.639722 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.639734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.641405 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.641410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.641598 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.641667 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.641795 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.641944 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":248,"threads_started":1}
I20260620 20:10:09.643793 11543 raft_consensus.cc:1203] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 20:10:09.644451 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.629167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.0.0.1:58592)
0620 20:10:09.629504 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.629529 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.629675 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.630171 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.630176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.630576 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.630713 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.630721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.633021 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.633027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.634111 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.634123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.634924 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.634930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.635137 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.635793 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.635817 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.638026 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.641563 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.641569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.641574 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.642028 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.642514 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.642519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.642523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.642693 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.643422 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.643427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.643591 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.643759 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.644113 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260620 20:10:09.644443 11571 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c832b7122d493794c7c2731fa72596. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } } }
I20260620 20:10:09.644716 11571 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:10:09.645334 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.628896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.4.169.189:36339)
0620 20:10:09.629403 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.629410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.629618 (+ 208us) server_negotiation.cc:407] Connection header received
0620 20:10:09.629885 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.629895 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.630003 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.630130 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.630856 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.632855 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.634242 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.634764 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.634907 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.638208 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.638219 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.638224 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.638262 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.641420 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.642153 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.642168 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.642171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.642348 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.642869 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.642875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.642883 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.643236 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.643367 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.644738 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.644962 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":797,"threads_started":1}
I20260620 20:10:09.646225 11649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986209, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.188@KRBTEST.COM for oryx/127.4.169.188@KRBTEST.COM
I20260620 20:10:09.658414 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.647049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:40759 (local address 127.0.0.1:52680)
0620 20:10:09.647222 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.647241 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.647344 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.647673 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.647677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.648043 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.648177 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.648184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.649091 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.649099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.649906 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.649915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.650667 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.650672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.650872 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.651389 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.651405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.653316 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.656081 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.656089 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.656094 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.656436 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.656774 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.656781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.656786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.656929 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.657418 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.657423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.657558 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.657688 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.657965 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 20:10:09.658440 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.647065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.4.169.188:40759)
0620 20:10:09.647222 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.647229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.647289 (+ 60us) server_negotiation.cc:407] Connection header received
0620 20:10:09.647455 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.647460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.647527 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.647660 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.648299 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.648973 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.650035 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.650527 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.650661 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.653530 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.653536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.653541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.653575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.655959 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.656542 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.656550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.656553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.656672 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.657027 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.657035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.657038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.657300 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.657443 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.657776 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.658094 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":60}
I20260620 20:10:09.663705 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Bootstrap starting.
I20260620 20:10:09.666900 11649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:09.667951 11649 log.cc:824] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:09.670881 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: No bootstrap required, opened a new log
I20260620 20:10:09.678021 11649 raft_consensus.cc:348] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } }
I20260620 20:10:09.678316 11649 raft_consensus.cc:374] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:09.678475 11649 raft_consensus.cc:729] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55538c2647943c1b75637fdb733692b, State: Initialized, Role: FOLLOWER
I20260620 20:10:09.679056 11649 consensus_queue.cc:260] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } }
I20260620 20:10:09.679927 11659 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } } }
I20260620 20:10:09.680303 11659 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:10:09.681540 11649 sys_catalog.cc:564] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:10:09.690524 11670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:10:09.690744 11670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:10:09.694001 11496 master_runner.cc:428] Detected that this master f55538c2647943c1b75637fdb733692b is joining an existing cluster
I20260620 20:10:09.694149 11496 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.169.188:40759
I20260620 20:10:09.730772 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.707291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.4.169.190:38587)
0620 20:10:09.707463 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.707471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.707605 (+ 134us) server_negotiation.cc:407] Connection header received
0620 20:10:09.707832 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.707839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.707935 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.708101 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.710339 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.711888 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.715325 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.716322 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.716458 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.722315 (+ 5857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.722327 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.722335 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.722409 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.726769 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.727940 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.727947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.727951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.728116 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.728645 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.728653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.728657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.729006 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.729165 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.730138 (+ 973us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.730419 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:10:09.733453 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.706707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:42382)
0620 20:10:09.707485 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.707509 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.707691 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.708707 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.708715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.709628 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.710096 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.710113 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.712176 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.712185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.714205 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.714222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.716915 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.716926 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.717197 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.719013 (+ 1816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.719047 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.720314 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.727250 (+ 6936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.727255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.727259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.727791 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.728274 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.728279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.728282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.728492 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.729254 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.729259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.729497 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.729985 (+ 488us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.731400 (+ 1415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":486,"threads_started":1}
I20260620 20:10:09.733918 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 10994
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.129@KRBTEST.COM
I20260620 20:10:09.750895 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.707986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.4.169.188:40759)
0620 20:10:09.708125 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.708130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.723516 (+ 15386us) server_negotiation.cc:407] Connection header received
0620 20:10:09.723754 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.723760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.723848 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.723986 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.733947 (+ 9961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.735057 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.738741 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.739539 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.739635 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.742632 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.742640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.742646 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.742680 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.746306 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.747675 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.747682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.747690 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.747898 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.748309 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.748314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.748320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.749067 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.749265 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.749959 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.750298 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:10:09.751171 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.706889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.4.169.189:36339)
0620 20:10:09.707111 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.707119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.724455 (+ 17336us) server_negotiation.cc:407] Connection header received
0620 20:10:09.724871 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.724878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.724984 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.725066 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.726978 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.728100 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.732934 (+ 4834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.733654 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.733781 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.742514 (+ 8733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.742523 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.742530 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.742579 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.747459 (+ 4880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.748320 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.748326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.748330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.748502 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.749396 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.749404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.749408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.749763 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.749910 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.750553 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.750813 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:10:09.752204 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.708322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.0.0.1:58616)
0620 20:10:09.724343 (+ 16021us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.724364 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.724489 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.725337 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.725345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.726438 (+ 1093us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.726788 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.726800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.728277 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.728282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.732740 (+ 4458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.732756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.738150 (+ 5394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.738160 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.738376 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.739302 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.739332 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.740626 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.747637 (+ 7011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.747648 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.747653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.748169 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.749073 (+ 904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.749078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.749081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.749262 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.750001 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.750008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.750236 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.751418 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.751745 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15861,"thread_start_us":18528,"threads_started":1}
I20260620 20:10:09.761915 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.734937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.4.169.129:35319)
0620 20:10:09.736262 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.736272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.736305 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:10:09.736376 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.736386 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.736679 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.738501 (+ 1822us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.738756 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.740317 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.742360 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.743370 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.744074 (+ 704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.753065 (+ 8991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.753083 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.753103 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.753162 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.757106 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.758045 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.758056 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.758060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.758226 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.758736 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.758744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.758750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.759200 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.759484 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.761226 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.761504 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":908,"threads_started":1}
I20260620 20:10:09.763561 11117 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.190@KRBTEST.COM for oryx/127.4.169.188@KRBTEST.COM
I20260620 20:10:09.765159 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.707767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:40759 (local address 127.0.0.1:52694)
0620 20:10:09.722947 (+ 15180us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.722979 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.723602 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.729477 (+ 5875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.729485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.731604 (+ 2119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.733048 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.733061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.736371 (+ 3310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.736379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.737900 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.737913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.739944 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.739950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.740142 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.741293 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.741335 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.742470 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.746891 (+ 4421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.746897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.746902 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.747530 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.748021 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.748029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.748034 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.748183 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.749335 (+ 1152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.749340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.749510 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.763918 (+ 14408us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.764470 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15002,"thread_start_us":16800,"threads_started":1}
I20260620 20:10:09.772264 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.744695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:40759 (local address 127.0.0.1:52706)
0620 20:10:09.746589 (+ 1894us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.746612 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.746720 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.752160 (+ 5440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.752167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.752627 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.752823 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.752835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.756103 (+ 3268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.756112 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.758014 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.758035 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.759515 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.759525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.759767 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.760549 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.760582 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.764985 (+ 4403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.769242 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.769248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.769253 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.769811 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.770321 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.770326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.770330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.770562 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.771250 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.771256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.771453 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.771603 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.771887 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1742}
I20260620 20:10:09.773411 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.745680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.4.169.188:40759)
0620 20:10:09.751689 (+ 6009us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.751696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.751735 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:10:09.751856 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.751864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.751987 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.754469 (+ 2482us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.754705 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.755898 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.758269 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.759298 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.763725 (+ 4427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.765222 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.765229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.765236 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.765279 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.769030 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.769972 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.769977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.769981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.770116 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.770709 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.770715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.770719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.771056 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.772625 (+ 1569us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.772751 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.772980 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5868,"thread_start_us":16878,"threads_started":1}
I20260620 20:10:09.777961 10933 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.169.188:40759: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f55538c2647943c1b75637fdb733692b"
member_type: NON_VOTER
last_known_addr {
host: "127.4.169.188"
port: 40759
}
attrs {
promote: true
}
}
dest_uuid: "e7c832b7122d493794c7c2731fa72596"
cas_config_opid_index: 5
I20260620 20:10:09.779330 10933 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } }
I20260620 20:10:09.781306 11654 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f55538c2647943c1b75637fdb733692b (127.4.169.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:09.787531 11624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:09.788086 11654 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.169.188:40759
I20260620 20:10:09.788632 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:09.788923 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:09.789151 11543 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 LEARNER]: Refusing update from remote peer e7c832b7122d493794c7c2731fa72596: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 20:10:09.789944 11682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 20:10:09.806936 11543 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f55538c2647943c1b75637fdb733692b (127.4.169.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:09.812114 11571 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: NON_VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: true } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:09.812422 11571 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:10:09.812868 11682 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: attempting to promote NON_VOTER dc5844a44fc04b1eb9479d71b25ace63 to VOTER
I20260620 20:10:09.813755 11682 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } }
I20260620 20:10:09.816510 11543 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 LEARNER]: Refusing update from remote peer e7c832b7122d493794c7c2731fa72596: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 20:10:09.817049 11683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 20:10:09.818991 11624 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Refusing update from remote peer e7c832b7122d493794c7c2731fa72596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 20:10:09.819550 11496 master.cc:537] Master@127.4.169.188:40759 shutting down...
I20260620 20:10:09.819535 11683 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dc5844a44fc04b1eb9479d71b25ace63 (127.4.169.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:09.820986 11623 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Refusing update from remote peer e7c832b7122d493794c7c2731fa72596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260620 20:10:09.821683 11603 tablet.cc:2369] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:10:09.823000 11654 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:09.823422 11542 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dc5844a44fc04b1eb9479d71b25ace63 (127.4.169.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:09.827713 11571 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:09.828409 11571 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:10:09.831202 11682 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:09.831516 11682 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:09.831991 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:09.832244 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:09.837867 11496 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:09.838366 11496 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:09.838640 11496 tablet_replica.cc:330] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: stopping tablet replica
W20260620 20:10:09.841670 10917 consensus_peers.cc:487] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 -> Peer f55538c2647943c1b75637fdb733692b (127.4.169.188:40759): Couldn't send request to peer f55538c2647943c1b75637fdb733692b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 20:10:09.846107 11496 master.cc:552] Master@127.4.169.188:40759 shutdown complete.
I20260620 20:10:09.846197 11496 master_runner.cc:305] Clearing existing system tablet
I20260620 20:10:09.846618 11496 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:09.846742 11496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:09.849620 11496 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:10:09.851222 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:09.851562 11496 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:09.851982 11496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:10:09.852133 11496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal
uuid: "f55538c2647943c1b75637fdb733692b"
format_stamp: "Formatted at 2026-06-20 20:10:09 on dist-test-slave-4bwq"
server_key: "0bb77102813ee4c8a93723262ba4e0e5"
server_key_iv: "8b7b15f98b484b803cb656eaffb2f0c2"
server_key_version: "encryptionkey@0"
I20260620 20:10:09.857349 11496 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:09.864425 11496 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:10:09.864698 11496 log.cc:1196] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 20:10:09.865234 11496 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Deleting consensus metadata
I20260620 20:10:09.865540 11496 master_runner.cc:315] Copying system tablet from 127.4.169.190:38587
I20260620 20:10:09.868177 11496 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.169.190:38587
I20260620 20:10:09.885021 11694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.869019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:42392)
0620 20:10:09.870767 (+ 1748us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:09.870789 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:09.870997 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:09.871501 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:09.871508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:09.872060 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:09.872315 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.872327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.873784 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.873790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.875290 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:09.875306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.876295 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.876302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:09.876485 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.877207 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:09.877230 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:09.878237 (+ 1007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:09.882024 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.882030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.882035 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.882550 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.883064 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:09.883070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:09.883074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:09.883248 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:09.883964 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:09.883969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:09.884149 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 20:10:09.884293 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 20:10:09.884607 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"thread_start_us":1490,"threads_started":1}
I20260620 20:10:09.885380 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:09.869073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.4.169.190:38587)
0620 20:10:09.869310 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:09.869317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:09.870886 (+ 1569us) server_negotiation.cc:407] Connection header received
0620 20:10:09.871152 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:09.871158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:09.871238 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:09.871309 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:09.872495 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.873628 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.875445 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:09.876117 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:09.876252 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:09.878507 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:09.878514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:09.878519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:09.878553 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:09.881867 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.882693 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.882699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.882702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.882900 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:09.883378 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:09.883383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:09.883387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:09.883758 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:09.883900 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:09.884475 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 20:10:09.884712 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 20:10:09.886006 10953 tablet_copy_service.cc:140] P e7c832b7122d493794c7c2731fa72596: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f55538c2647943c1b75637fdb733692b ({username='oryx', principal='oryx/127.4.169.188@KRBTEST.COM'} at 127.0.0.1:42392)
I20260620 20:10:09.886170 10953 tablet_copy_service.cc:161] P e7c832b7122d493794c7c2731fa72596: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f55538c2647943c1b75637fdb733692b at {username='oryx', principal='oryx/127.4.169.188@KRBTEST.COM'} at 127.0.0.1:42392: session id = f55538c2647943c1b75637fdb733692b-00000000000000000000000000000000
I20260620 20:10:09.890496 10953 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:10:09.892504 11496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:10:09.899530 11496 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: tablet copy: Starting download of 0 data blocks...
I20260620 20:10:09.899832 11496 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: tablet copy: Starting download of 1 WAL segments...
I20260620 20:10:09.902432 11496 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:10:09.905421 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 10994
I20260620 20:10:09.908089 10953 tablet_copy_service.cc:342] P e7c832b7122d493794c7c2731fa72596: Request end of tablet copy session f55538c2647943c1b75637fdb733692b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.169.188@KRBTEST.COM'} at 127.0.0.1:42392
I20260620 20:10:09.908239 10953 tablet_copy_service.cc:433] P e7c832b7122d493794c7c2731fa72596: ending tablet copy session f55538c2647943c1b75637fdb733692b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f55538c2647943c1b75637fdb733692b
I20260620 20:10:09.920877 11496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:09.923779 11695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:09.927268 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:35319
--local_ip_for_outbound_sockets=127.4.169.129
--tserver_master_addrs=127.4.169.190:38587
--webserver_port=46537
--webserver_interface=127.4.169.129
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab
--principal=oryx/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.169.190:38587,127.4.169.189:36339
--tserver_master_addrs=127.4.169.190:38587,127.4.169.189:36339,127.4.169.188:40759 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:09.927649 11696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:09.935766 11496 server_base.cc:1034] running on GCE node
W20260620 20:10:09.936822 11698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:09.937347 11496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:09.938059 11496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:09.939265 11496 hybrid_clock.cc:648] HybridClock initialized: now 1781986209939224 us; error 27 us; skew 500 ppm
I20260620 20:10:09.939507 11496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:09 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986209, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:09.941783 11496 init.cc:377] Logged in from keytab as oryx/127.4.169.188@KRBTEST.COM (short username oryx)
I20260620 20:10:09.943193 11496 webserver.cc:460] Webserver started at http://127.4.169.188:40579/ using document root <none> and password file <none>
I20260620 20:10:09.943557 11496 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:09.943665 11496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:09.947054 11496 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:10:09.949406 11706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:09.949893 11496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:10:09.950030 11496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal
uuid: "f55538c2647943c1b75637fdb733692b"
format_stamp: "Formatted at 2026-06-20 20:10:09 on dist-test-slave-4bwq"
server_key: "0bb77102813ee4c8a93723262ba4e0e5"
server_key_iv: "8b7b15f98b484b803cb656eaffb2f0c2"
server_key_version: "encryptionkey@0"
I20260620 20:10:09.950275 11496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:09.969453 11496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:09.973420 11496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:10.011425 11496 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.188:40759
I20260620 20:10:10.011475 11757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.188:40759 every 8 connection(s)
I20260620 20:10:10.012521 11496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
I20260620 20:10:10.018255 11758 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:10:10.022663 11758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Bootstrap starting.
I20260620 20:10:10.043742 11758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:10:10.044307 11758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: Bootstrap complete.
I20260620 20:10:10.045389 11758 raft_consensus.cc:348] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } }
I20260620 20:10:10.045711 11758 raft_consensus.cc:729] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f55538c2647943c1b75637fdb733692b, State: Initialized, Role: LEARNER
I20260620 20:10:10.045995 11758 consensus_queue.cc:260] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } }
I20260620 20:10:10.046891 11761 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:10.047356 11761 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: This master's current role is: LEARNER
I20260620 20:10:10.047209 11758 sys_catalog.cc:564] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:10.058970 11772 catalog_manager.cc:1259] Loaded cluster ID: db37cee9ee884b908d24568ad4881b07
I20260620 20:10:10.059098 11772 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: loading cluster ID for follower catalog manager: success
I20260620 20:10:10.061498 11772 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: acquiring CA information for follower catalog manager: success
I20260620 20:10:10.063354 11772 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:10:10.290179 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:10.274920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:40759 (local address 127.0.0.1:52710)
0620 20:10:10.283051 (+ 8131us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:10.283078 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:10.283292 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:10.283758 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:10.283762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:10.283789 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:10.284007 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:10.284016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.285868 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.285872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:10.287379 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:10.287393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.289450 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.289455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:10.289601 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:10.289628 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:10:10.289714 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 20:10:10.289816 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8011,"thread_start_us":7905,"threads_started":1}
I20260620 20:10:10.290220 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:10.275116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.4.169.188:40759)
0620 20:10:10.282901 (+ 7785us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:10.282908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:10.283192 (+ 284us) server_negotiation.cc:407] Connection header received
0620 20:10:10.283445 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:10.283453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:10.283551 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:10.283717 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:10.284181 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.285711 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.287534 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.289237 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.289380 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:10.289590 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:10.289763 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 20:10:10.289908 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7626,"thread_start_us":7453,"threads_started":1}
I20260620 20:10:10.291720 11733 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260620 20:10:10.292344 11761 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c832b7122d493794c7c2731fa72596. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:10.292646 11761 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: This master's current role is: LEARNER
W20260620 20:10:10.334954 11700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:10.335240 11700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:10.335333 11700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:10.343071 11700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:10.343189 11700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:10.343302 11700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:10:10.353300 11700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab
--principal=oryx/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:35319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=46537
--tserver_master_addrs=127.4.169.190:38587,127.4.169.189:36339,127.4.169.188:40759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:10.354780 11700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:10.356114 11700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:10.369814 11779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:10.369753 11782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:10.369853 11780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:10.370839 11700 server_base.cc:1034] running on GCE node
I20260620 20:10:10.371379 11700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:10.372247 11700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:10.373468 11700 hybrid_clock.cc:648] HybridClock initialized: now 1781986210373420 us; error 43 us; skew 500 ppm
I20260620 20:10:10.373792 11700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:10 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986210, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:10.377033 11700 init.cc:377] Logged in from keytab as oryx/127.4.169.129@KRBTEST.COM (short username oryx)
I20260620 20:10:10.378543 11700 webserver.cc:460] Webserver started at http://127.4.169.129:46537/ using document root <none> and password file <none>
I20260620 20:10:10.379690 11700 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:10.379863 11700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:10.385123 11700 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:10:10.387912 11789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:10.389407 11700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:10:10.389606 11700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "c3e85269446e48feb97c9899cc44b756"
format_stamp: "Formatted at 2026-06-20 20:10:07 on dist-test-slave-4bwq"
server_key: "8707ec96d54bfadf40092f5d4e73b0d4"
server_key_iv: "992ae38fafd72aab6dd1e5a879d24af8"
server_key_version: "encryptionkey@0"
I20260620 20:10:10.390687 11700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:10.411607 11700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:10.417158 11700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:10.417461 11700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:10.419028 11700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:10.420552 11700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:10.420684 11700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:10.420830 11700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:10.420926 11700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:10.458585 11700 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:35319
I20260620 20:10:10.458608 11901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:35319 every 8 connection(s)
I20260620 20:10:10.460058 11700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:10:10.467463 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 11700
I20260620 20:10:10.467794 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 11128
Jun 20 20:10:10 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986210, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.188@KRBTEST.COM
Jun 20 20:10:10 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986210, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.189@KRBTEST.COM
I20260620 20:10:10.490653 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:10.463557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:60255 (local address 127.4.169.189:36339)
0620 20:10:10.464089 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:10.464094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:10.465751 (+ 1657us) server_negotiation.cc:407] Connection header received
0620 20:10:10.468617 (+ 2866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:10.468623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:10.468740 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:10.468838 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:10.472023 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.473193 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.476291 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.476949 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.477049 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:10.483481 (+ 6432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:10.483489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:10.483495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:10.483542 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:10.487048 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:10.487896 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:10.487902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:10.487906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:10.488044 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:10.488526 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:10.488531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:10.488534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:10.488896 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:10.489051 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:10.490038 (+ 987us) server_negotiation.cc:299] Negotiation successful
0620 20:10:10.490239 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":356,"threads_started":1}
I20260620 20:10:10.491492 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:10.463338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:40693 (local address 127.4.169.188:40759)
0620 20:10:10.463519 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:10.463526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:10.465752 (+ 2226us) server_negotiation.cc:407] Connection header received
0620 20:10:10.468617 (+ 2865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:10.468623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:10.468721 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:10.468799 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:10.472023 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.473199 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.476291 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.476945 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.477049 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:10.482979 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:10.482985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:10.482989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:10.483024 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:10.486807 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:10.487977 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:10.487985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:10.487988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:10.488136 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:10.488599 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:10.488606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:10.488611 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:10.488909 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:10.489045 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:10.491019 (+ 1974us) server_negotiation.cc:299] Negotiation successful
0620 20:10:10.491259 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:10:10.492302 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:10.463938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.4.169.129:60255)
0620 20:10:10.465539 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:10.465617 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:10.468316 (+ 2699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:10.469411 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:10.469423 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:10.470560 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:10.471791 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:10.471819 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.473418 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.473423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:10.476109 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:10.476125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.477415 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.477423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:10.477890 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:10.478905 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:10.478951 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:10.483231 (+ 4280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:10.487221 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:10.487226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:10.487236 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:10.487768 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:10.488183 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:10.488189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:10.488194 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:10.488400 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:10.489039 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:10.489051 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:10.489550 (+ 499us) client_negotiation.cc:769] Sending connection context
0620 20:10:10.489940 (+ 390us) client_negotiation.cc:240] Negotiation successful
0620 20:10:10.490594 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":412,"spinlock_wait_cycles":67456,"thread_start_us":493,"threads_started":1}
I20260620 20:10:10.492681 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:10.463938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:40759 (local address 127.4.169.129:40693)
0620 20:10:10.465539 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:10.465617 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:10.468164 (+ 2547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:10.469411 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:10.469423 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:10.470560 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:10.471791 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:10.471819 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.473435 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.473439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:10.476117 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:10.476125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.477415 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.477423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:10.477886 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:10.478983 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:10.479000 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:10.482716 (+ 3716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:10.486992 (+ 4276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:10.487004 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:10.487027 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:10.487853 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:10.488253 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:10.488258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:10.488262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:10.488484 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:10.490503 (+ 2019us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:10.490508 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:10.490710 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 20:10:10.491767 (+ 1057us) client_negotiation.cc:240] Negotiation successful
0620 20:10:10.492116 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":603,"spinlock_wait_cycles":27136,"thread_start_us":435,"threads_started":1}
I20260620 20:10:10.494094 11904 heartbeater.cc:347] Connected to a master server at 127.4.169.189:36339
I20260620 20:10:10.494515 11904 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:10.495528 11904 heartbeater.cc:511] Master 127.4.169.189:36339 requested a full tablet report, sending...
I20260620 20:10:10.498283 11533 ts_manager.cc:194] Registered new tserver with Master: c3e85269446e48feb97c9899cc44b756 (127.4.169.129:35319)
Jun 20 20:10:10 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986210, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:10.503334 11902 heartbeater.cc:347] Connected to a master server at 127.4.169.188:40759
I20260620 20:10:10.503489 11902 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:10.503916 11902 heartbeater.cc:511] Master 127.4.169.188:40759 requested a full tablet report, sending...
I20260620 20:10:10.505545 11723 ts_manager.cc:194] Registered new tserver with Master: c3e85269446e48feb97c9899cc44b756 (127.4.169.129:35319)
I20260620 20:10:10.508847 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:10.466149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:57801 (local address 127.4.169.190:38587)
0620 20:10:10.481825 (+ 15676us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:10.481834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:10.493444 (+ 11610us) server_negotiation.cc:407] Connection header received
0620 20:10:10.493531 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:10.493538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:10.493639 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:10.493773 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:10.494737 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.495760 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.497246 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.497681 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.497825 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:10.502078 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:10.502087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:10.502093 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:10.502143 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:10.505875 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:10.506718 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:10.506725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:10.506730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:10.506887 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:10.507278 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:10.507288 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:10.507292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:10.507722 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:10.507901 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:10.508335 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 20:10:10.508594 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15515,"thread_start_us":15682,"threads_started":1}
I20260620 20:10:10.508930 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:10.465912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.129:57801)
0620 20:10:10.493069 (+ 27157us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:10.493088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:10.493194 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:10.493798 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:10.493803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:10.494349 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:10.494588 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:10.494601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.495899 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.495905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:10.497100 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:10.497118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.497875 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.497880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:10.498059 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:10.499257 (+ 1198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:10.499273 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:10.501851 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:10.505997 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:10.506002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:10.506006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:10.506573 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:10.507029 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:10.507034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:10.507037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:10.507180 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:10.507874 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:10.507877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:10.508068 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 20:10:10.508220 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 20:10:10.508589 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27038,"thread_start_us":36247,"threads_started":1}
I20260620 20:10:10.510093 11903 heartbeater.cc:347] Connected to a master server at 127.4.169.190:38587
I20260620 20:10:10.510243 11903 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:10.510677 11903 heartbeater.cc:511] Master 127.4.169.190:38587 requested a full tablet report, sending...
I20260620 20:10:10.511761 10933 ts_manager.cc:194] Re-registered known tserver with Master: c3e85269446e48feb97c9899cc44b756 (127.4.169.129:35319)
I20260620 20:10:10.512529 10933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:57801
Jun 20 20:10:10 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.130@KRBTEST.COM
I20260620 20:10:10.520357 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:10.469376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.4.169.130:41699)
0620 20:10:10.499059 (+ 29683us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:10.499068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:10.501315 (+ 2247us) server_negotiation.cc:407] Connection header received
0620 20:10:10.501425 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:10.501440 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:10.501836 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:10.502035 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:10.504173 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.505928 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.508056 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:10.509078 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:10.509209 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:10.514537 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:10.514550 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:10.514564 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:10.514598 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:10.517651 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:10.518308 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:10.518315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:10.518317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:10.518500 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:10.518825 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:10.518829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:10.518831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:10.519137 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:10.519356 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:10.519698 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 20:10:10.519980 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29498,"thread_start_us":29318,"threads_started":1}
I20260620 20:10:10.521551 11251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:10.604441 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 11128
I20260620 20:10:10.620769 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:41699
--local_ip_for_outbound_sockets=127.4.169.130
--tserver_master_addrs=127.4.169.190:38587
--webserver_port=37863
--webserver_interface=127.4.169.130
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab
--principal=oryx/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.169.190:38587,127.4.169.189:36339
--tserver_master_addrs=127.4.169.190:38587,127.4.169.189:36339,127.4.169.188:40759 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:10:10.772120 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: NON_VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: true } } }
I20260620 20:10:10.772459 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:10.787400 11915 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: attempting to promote NON_VOTER f55538c2647943c1b75637fdb733692b to VOTER
I20260620 20:10:10.788209 11915 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } }
I20260620 20:10:10.790321 11543 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Refusing update from remote peer e7c832b7122d493794c7c2731fa72596: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 20:10:10.790342 11733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 LEARNER]: Refusing update from remote peer e7c832b7122d493794c7c2731fa72596: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 20:10:10.790998 11915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 20:10:10.791379 11916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 20:10:10.794955 11916 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f55538c2647943c1b75637fdb733692b (127.4.169.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:10.795617 11543 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f55538c2647943c1b75637fdb733692b (127.4.169.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:10.798914 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:10.799253 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:10.799511 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:10.799700 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:10.799258 11916 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:10.800129 11916 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:10.800136 11733 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f55538c2647943c1b75637fdb733692b (127.4.169.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:10.800611 11919 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:10.800935 11919 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:10:10.807302 11920 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7c832b7122d493794c7c2731fa72596" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:10.807642 11920 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 20:10:10.960245 11914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:10.960536 11914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:10.960631 11914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:10.968055 11914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:10.968212 11914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:10.968346 11914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:10:10.978211 11914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab
--principal=oryx/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:41699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=37863
--tserver_master_addrs=127.4.169.190:38587,127.4.169.189:36339,127.4.169.188:40759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:10.979487 11914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:10.980721 11914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:10.994349 11926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:10.994439 11925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:10.994270 11928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:10.996035 11914 server_base.cc:1034] running on GCE node
I20260620 20:10:10.996716 11914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:10.997668 11914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:10.998867 11914 hybrid_clock.cc:648] HybridClock initialized: now 1781986210998818 us; error 36 us; skew 500 ppm
I20260620 20:10:10.999163 11914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:10 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986210, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:11.002671 11914 init.cc:377] Logged in from keytab as oryx/127.4.169.130@KRBTEST.COM (short username oryx)
I20260620 20:10:11.004189 11914 webserver.cc:460] Webserver started at http://127.4.169.130:37863/ using document root <none> and password file <none>
I20260620 20:10:11.004944 11914 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:11.005108 11914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:11.011312 11914 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:10:11.014228 11935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:11.015900 11914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:10:11.016120 11914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "f359aa5fee0d4e49949dc64a62e96dbb"
format_stamp: "Formatted at 2026-06-20 20:10:07 on dist-test-slave-4bwq"
server_key: "0d3ae442c022227b1b5ce53351b1bc4e"
server_key_iv: "8c81a5b51fe858be179d3eb6954979c6"
server_key_version: "encryptionkey@0"
I20260620 20:10:11.017287 11914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:11.032791 11914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:11.037822 11914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:11.038126 11914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:11.039172 11914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:11.040517 11914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:11.040630 11914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:11.040735 11914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:11.040827 11914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:11.073052 11914 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:41699
I20260620 20:10:11.073143 12047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:41699 every 8 connection(s)
I20260620 20:10:11.074610 11914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:10:11.083386 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 11914
I20260620 20:10:11.083694 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 11262
Jun 20 20:10:11 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.131@KRBTEST.COM
Jun 20 20:10:11 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986210, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
Jun 20 20:10:11 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986210, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.189@KRBTEST.COM
Jun 20 20:10:11 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986210, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.188@KRBTEST.COM
I20260620 20:10:11.127952 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.085490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.4.169.131:40573)
0620 20:10:11.089663 (+ 4173us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.089676 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.089715 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:10:11.089804 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.089817 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.090202 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.090426 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.101037 (+ 10611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.103639 (+ 2602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.108369 (+ 4730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.109843 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.110019 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.117688 (+ 7669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.117708 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.117728 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.117776 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.123156 (+ 5380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.124140 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.124151 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.124155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.124774 (+ 619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.125776 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.125787 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.125792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.126235 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.126582 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.127217 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.127513 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3942,"thread_start_us":3751,"threads_started":1}
I20260620 20:10:11.129390 11385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:11.131898 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.077489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:36083 (local address 127.4.169.190:38587)
0620 20:10:11.093063 (+ 15574us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.093070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.094942 (+ 1872us) server_negotiation.cc:407] Connection header received
0620 20:10:11.101195 (+ 6253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.101201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.101711 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.101826 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.112771 (+ 10945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.114105 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.116621 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.117362 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.117504 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.122095 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.122104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.122111 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.122163 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.126485 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.129352 (+ 2867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.129359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.129363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.129506 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.130532 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.130541 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.130543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.130789 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.130951 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.131339 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.131562 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15438,"thread_start_us":15575,"threads_started":1}
I20260620 20:10:11.133061 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.079059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.4.169.130:42227)
0620 20:10:11.098402 (+ 19343us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.098427 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.098569 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.110138 (+ 11569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.110153 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.111303 (+ 1150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:11.112555 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.112587 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.113731 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.113736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.116277 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.116290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.117196 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.117203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.117572 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.118763 (+ 1191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:11.118787 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:11.122568 (+ 3781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:11.126649 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.126660 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.126679 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.127230 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.128983 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.128986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.128989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.129163 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.129933 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:11.129939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:11.130239 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.131049 (+ 810us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.132340 (+ 1291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19143,"mutex_wait_us":402,"spinlock_wait_cycles":1408,"thread_start_us":18947,"threads_started":1}
I20260620 20:10:11.133724 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.077516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:42227 (local address 127.4.169.189:36339)
0620 20:10:11.109460 (+ 31944us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.109465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.109484 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:10:11.109545 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.109549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.109612 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.109770 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.112795 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.113525 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.116467 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.116951 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.117088 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.122774 (+ 5686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.122779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.122782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.122827 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.126492 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.128192 (+ 1700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.128197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.128200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.128361 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.129292 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.129300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.129307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.129709 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.129875 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.133136 (+ 3261us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.133375 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31829,"thread_start_us":32370,"threads_started":1}
I20260620 20:10:11.133852 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.095196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:40759 (local address 127.4.169.130:47839)
0620 20:10:11.107091 (+ 11895us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.107115 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.107259 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.112667 (+ 5408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.114297 (+ 1630us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.115053 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:11.115473 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.115486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.118864 (+ 3378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.118871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.120659 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.120674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.121865 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.121874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.123060 (+ 1186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.123574 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:11.123598 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:11.126380 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:11.130286 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.130294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.130301 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.130983 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.131521 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.131529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.131536 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.131754 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.132559 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:11.132568 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:11.132782 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.132966 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.133295 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11720,"thread_start_us":11536,"threads_started":1}
I20260620 20:10:11.134240 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.078631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.130:36083)
0620 20:10:11.093923 (+ 15292us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.093995 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.097372 (+ 3377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.110138 (+ 12766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.110153 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.111303 (+ 1150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:11.112555 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.112587 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.114265 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.114271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.116276 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.116290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.117553 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.117559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.117715 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.118440 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:11.118483 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:11.121834 (+ 3351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:11.127369 (+ 5535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.127373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.127376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.127982 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.129652 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.129657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.129661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.129815 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.130905 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:11.130910 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:11.131092 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.131217 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.131615 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14903,"mutex_wait_us":189,"thread_start_us":14662,"threads_started":1}
I20260620 20:10:11.135015 12050 heartbeater.cc:347] Connected to a master server at 127.4.169.189:36339
I20260620 20:10:11.135301 12050 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:11.136022 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.095229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:47839 (local address 127.4.169.188:40759)
0620 20:10:11.112184 (+ 16955us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.112190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.112212 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:10:11.112276 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.112282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.112366 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.115641 (+ 3275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.115842 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.118675 (+ 2833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.120887 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.121614 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.122512 (+ 898us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.126641 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.126647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.126652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.126701 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.130098 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.131191 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.131196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.131202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.131375 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.131918 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.131926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.131932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.132380 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.134934 (+ 2554us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.135103 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.135433 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16810,"thread_start_us":17243,"threads_started":1}
I20260620 20:10:11.136070 12050 heartbeater.cc:511] Master 127.4.169.189:36339 requested a full tablet report, sending...
I20260620 20:10:11.136888 12048 heartbeater.cc:347] Connected to a master server at 127.4.169.188:40759
I20260620 20:10:11.137022 12048 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:11.137523 12048 heartbeater.cc:511] Master 127.4.169.188:40759 requested a full tablet report, sending...
I20260620 20:10:11.137715 11533 ts_manager.cc:194] Registered new tserver with Master: f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699)
I20260620 20:10:11.138486 12049 heartbeater.cc:347] Connected to a master server at 127.4.169.190:38587
I20260620 20:10:11.138635 12049 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:11.138773 11723 ts_manager.cc:194] Registered new tserver with Master: f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699)
I20260620 20:10:11.139508 12049 heartbeater.cc:511] Master 127.4.169.190:38587 requested a full tablet report, sending...
I20260620 20:10:11.140444 10933 ts_manager.cc:194] Re-registered known tserver with Master: f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699)
I20260620 20:10:11.141309 10933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:36083
I20260620 20:10:11.227387 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 11262
I20260620 20:10:11.243717 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:40573
--local_ip_for_outbound_sockets=127.4.169.131
--tserver_master_addrs=127.4.169.190:38587
--webserver_port=43147
--webserver_interface=127.4.169.131
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab
--principal=oryx/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.169.190:38587,127.4.169.189:36339
--tserver_master_addrs=127.4.169.190:38587,127.4.169.189:36339,127.4.169.188:40759 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:10:11.515084 11903 heartbeater.cc:503] Master 127.4.169.190:38587 was elected leader, sending a full tablet report...
W20260620 20:10:11.581892 12061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:11.582165 12061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:11.582357 12061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:11.589619 12061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:11.589722 12061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:11.589908 12061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:10:11.600210 12061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab
--principal=oryx/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:40573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=43147
--tserver_master_addrs=127.4.169.190:38587,127.4.169.189:36339,127.4.169.188:40759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:11.601789 12061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:11.603425 12061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:11.616564 12066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:11.616657 12069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:11.616725 12067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:11.617843 12061 server_base.cc:1034] running on GCE node
I20260620 20:10:11.618476 12061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:11.619346 12061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:11.620549 12061 hybrid_clock.cc:648] HybridClock initialized: now 1781986211620493 us; error 48 us; skew 500 ppm
I20260620 20:10:11.620820 12061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:11 dist-test-slave-4bwq krb5kdc[10888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986211, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:11.624123 12061 init.cc:377] Logged in from keytab as oryx/127.4.169.131@KRBTEST.COM (short username oryx)
I20260620 20:10:11.625613 12061 webserver.cc:460] Webserver started at http://127.4.169.131:43147/ using document root <none> and password file <none>
I20260620 20:10:11.626314 12061 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:11.626533 12061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:11.631705 12061 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 20:10:11.634686 12076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:11.636078 12061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:10:11.636204 12061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "23d7e639f3ee49efab35348707f2a64e"
format_stamp: "Formatted at 2026-06-20 20:10:08 on dist-test-slave-4bwq"
server_key: "69b91401d92913bd01c273b1077b1ee3"
server_key_iv: "6c9e5ff6d7f9040527be4ade21ccb939"
server_key_version: "encryptionkey@0"
I20260620 20:10:11.637223 12061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:11.669344 12061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:11.673434 12061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:11.673735 12061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:11.674867 12061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:11.676147 12061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:11.676236 12061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:11.676312 12061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:11.676371 12061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:11.709021 12061 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:40573
I20260620 20:10:11.709115 12188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:40573 every 8 connection(s)
I20260620 20:10:11.710491 12061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:10:11.715509 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 12061
Jun 20 20:10:11 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986211, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.189@KRBTEST.COM
Jun 20 20:10:11 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986211, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.188@KRBTEST.COM
I20260620 20:10:11.771003 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.713414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:51261 (local address 127.4.169.189:36339)
0620 20:10:11.726424 (+ 13010us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.726434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.736610 (+ 10176us) server_negotiation.cc:407] Connection header received
0620 20:10:11.739350 (+ 2740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.739359 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.739487 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.739584 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.746369 (+ 6785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.747413 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.752207 (+ 4794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.752881 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.752990 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.760905 (+ 7915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.760927 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.760934 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.761002 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.766301 (+ 5299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.767308 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.767317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.767323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.767520 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.768172 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.768180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.768186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.768616 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.768790 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.770092 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.770407 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12796,"thread_start_us":12652,"threads_started":1}
I20260620 20:10:11.774549 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.714287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.4.169.131:51261)
0620 20:10:11.736472 (+ 22185us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.736499 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.739002 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.739884 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.739892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.741152 (+ 1260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:11.745249 (+ 4097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.745266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.748894 (+ 3628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.748902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.751598 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.751612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.753337 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.753346 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.753705 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.756215 (+ 2510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:11.756246 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:11.760620 (+ 4374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:11.766493 (+ 5873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.766499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.766503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.767132 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.767707 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.767714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.767720 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.768018 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.768901 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:11.768914 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:11.769546 (+ 632us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.771796 (+ 2250us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.772585 (+ 789us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21967,"mutex_wait_us":270,"thread_start_us":12309,"threads_started":1}
I20260620 20:10:11.776517 12191 heartbeater.cc:347] Connected to a master server at 127.4.169.189:36339
I20260620 20:10:11.776835 12191 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:11.777679 12191 heartbeater.cc:511] Master 127.4.169.189:36339 requested a full tablet report, sending...
I20260620 20:10:11.779374 11533 ts_manager.cc:194] Registered new tserver with Master: 23d7e639f3ee49efab35348707f2a64e (127.4.169.131:40573)
I20260620 20:10:11.783501 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.730916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.4.169.190:38587)
0620 20:10:11.752177 (+ 21261us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.752186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.752222 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:10:11.752296 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.752304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.752404 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.752548 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.757202 (+ 4654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.758333 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.768101 (+ 9768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.768786 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.769478 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.771957 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.771963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.771968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.772005 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.778167 (+ 6162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.779290 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.779298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.779310 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.779616 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.780227 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.780235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.780240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.780761 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.781351 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.782648 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.782965 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21093,"thread_start_us":21758,"threads_started":1}
I20260620 20:10:11.785701 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.713576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:48071 (local address 127.4.169.188:40759)
0620 20:10:11.728484 (+ 14908us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.728489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.728508 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:10:11.732459 (+ 3951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.732470 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.732607 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.732722 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.744891 (+ 12169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.746204 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.754636 (+ 8432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.755256 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.755345 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.761651 (+ 6306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.761668 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.761674 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.761719 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.765765 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.782695 (+ 16930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.782702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.782706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.782877 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.783372 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.783377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.783381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.783730 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.783890 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.785073 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.785326 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14799,"thread_start_us":14686,"threads_started":1}
I20260620 20:10:11.786815 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.714286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:40759 (local address 127.4.169.131:48071)
0620 20:10:11.727533 (+ 13247us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.727601 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.730344 (+ 2743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.733514 (+ 3170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.733532 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.736973 (+ 3441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:11.738083 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.738117 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.747700 (+ 9583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.747706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.752058 (+ 4352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.752070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.755605 (+ 3535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.755610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.755813 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.757866 (+ 2053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:11.757888 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:11.761423 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:11.765942 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.765951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.765971 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.782518 (+ 16547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.783036 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.783044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.783049 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.783235 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.783987 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:11.783994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:11.784175 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.785930 (+ 1755us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.786365 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12800,"thread_start_us":21552,"threads_started":1}
I20260620 20:10:11.787725 12189 heartbeater.cc:347] Connected to a master server at 127.4.169.188:40759
I20260620 20:10:11.787866 12189 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:11.788347 12189 heartbeater.cc:511] Master 127.4.169.188:40759 requested a full tablet report, sending...
I20260620 20:10:11.789579 11723 ts_manager.cc:194] Registered new tserver with Master: 23d7e639f3ee49efab35348707f2a64e (127.4.169.131:40573)
I20260620 20:10:11.794131 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.731226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.4.169.189:36339)
0620 20:10:11.743845 (+ 12619us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.743854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.745310 (+ 1456us) server_negotiation.cc:407] Connection header received
0620 20:10:11.745366 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.745371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.745442 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.745537 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.754536 (+ 8999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.755625 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.772836 (+ 17211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.773485 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.773572 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.783137 (+ 9565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.783145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.783150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.783196 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.786522 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.790038 (+ 3516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.790043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.790047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.790235 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.792674 (+ 2439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.792679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.792682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.792981 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.793211 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.793504 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.793718 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12447,"thread_start_us":12280,"threads_started":1}
Jun 20 20:10:11 dist-test-slave-4bwq krb5kdc[10888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986211, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:11.798818 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.741909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.4.169.188:40759)
0620 20:10:11.746083 (+ 4174us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.746092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.746358 (+ 266us) server_negotiation.cc:407] Connection header received
0620 20:10:11.746471 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.746479 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.746577 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.746712 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.749016 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.750124 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.751890 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.756379 (+ 4489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.756486 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.761058 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.761065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.761070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.761110 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.784782 (+ 23672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.788712 (+ 3930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.788717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.788720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.788861 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.789580 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.789584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.789588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.789904 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.791635 (+ 1731us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.791758 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.792017 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3994,"thread_start_us":5592,"threads_started":1}
I20260620 20:10:11.801566 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.736812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:49337)
0620 20:10:11.759515 (+ 22703us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.759537 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.759666 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.784623 (+ 24957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.784631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.785421 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:11.786030 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.786049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.788278 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.788283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.790229 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.790246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.791328 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.791335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.791504 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.792509 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:11.792531 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:11.795226 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:11.799104 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.799108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.799112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.799596 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.800062 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:11.800068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:11.800072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:11.800231 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:11.800876 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:11.800879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:11.801017 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.801128 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.801315 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22561,"thread_start_us":22404,"threads_started":1}
I20260620 20:10:11.801851 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.752885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:49337 (local address 127.4.169.190:38587)
0620 20:10:11.784111 (+ 31226us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.784119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.784143 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:10:11.784203 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.784211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.784304 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.784423 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:11.786232 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.788096 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.790373 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.791136 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.791267 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.795434 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:11.795444 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:11.795449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:11.795485 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:11.798948 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.799773 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.799779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.799782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.799928 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:11.800374 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:11.800379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:11.800382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:11.800742 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:11.800887 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.801247 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.801461 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31080,"thread_start_us":34918,"threads_started":1}
I20260620 20:10:11.802713 12190 heartbeater.cc:347] Connected to a master server at 127.4.169.190:38587
I20260620 20:10:11.802860 12190 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:11.803277 12190 heartbeater.cc:511] Master 127.4.169.190:38587 requested a full tablet report, sending...
I20260620 20:10:11.804216 10932 ts_manager.cc:194] Re-registered known tserver with Master: 23d7e639f3ee49efab35348707f2a64e (127.4.169.131:40573)
I20260620 20:10:11.793118 10933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:10:11.805127 10932 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:49337
W20260620 20:10:11.806219 10933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:10:11.857364 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.837413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.4.169.129:35319)
0620 20:10:11.846036 (+ 8623us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.846050 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.848665 (+ 2615us) server_negotiation.cc:407] Connection header received
0620 20:10:11.848804 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.848809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.848983 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.849133 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:11.849846 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.851673 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.854933 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.856293 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.856435 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.856648 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.856905 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.857053 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8389,"thread_start_us":8225,"threads_started":1}
I20260620 20:10:11.857461 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.837026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:35319 (local address 127.0.0.1:57034)
0620 20:10:11.848410 (+ 11384us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.848429 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.848542 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.849159 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.849166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.849202 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:11.849657 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.849671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.851841 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.851846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.854785 (+ 2939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.854797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.856518 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.856524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.856703 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.856738 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.856848 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.856979 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11258,"mutex_wait_us":32,"thread_start_us":7953,"threads_started":1}
I20260620 20:10:11.857797 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.835107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:40573 (local address 127.0.0.1:43964)
0620 20:10:11.844806 (+ 9699us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.844829 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.844991 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.847725 (+ 2734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.847734 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.847788 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:11.848499 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.848518 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.851399 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.851404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.853759 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.853770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.856639 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.856645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.856820 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.856854 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.856941 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.857075 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9544,"mutex_wait_us":28,"thread_start_us":12987,"threads_started":1}
I20260620 20:10:11.858512 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.835694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.4.169.131:40573)
0620 20:10:11.846721 (+ 11027us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.846733 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.846777 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:10:11.846941 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.846954 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.847346 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.847615 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:11.849411 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.851022 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.853927 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.855611 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.855733 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.855943 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.858027 (+ 2084us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.858182 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10799,"thread_start_us":10577,"threads_started":1}
I20260620 20:10:11.859911 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.835531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:41699 (local address 127.0.0.1:53412)
0620 20:10:11.845294 (+ 9763us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.845315 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.845441 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.852082 (+ 6641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.852087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.852115 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:11.852532 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.852541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.854417 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.854423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.856370 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.856381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.858352 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.858358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.858575 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.858615 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.859469 (+ 854us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.859598 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9586,"thread_start_us":8966,"threads_started":1}
I20260620 20:10:11.860122 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.835713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.4.169.130:41699)
0620 20:10:11.851478 (+ 15765us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.851486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.851509 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:10:11.851587 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.851594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.851834 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.852049 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:11.852727 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.854251 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.856529 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.858136 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.858304 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.858521 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.859698 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.859855 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15617,"thread_start_us":15540,"threads_started":1}
I20260620 20:10:11.861761 12124 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7a0b0d2b8e484b8b515059bdb0cf67 (DEFAULT_TABLE table=test-table [id=dae21e5ff7084ea4945078bd87354553]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:11.861598 11837 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7a0b0d2b8e484b8b515059bdb0cf67 (DEFAULT_TABLE table=test-table [id=dae21e5ff7084ea4945078bd87354553]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:11.863095 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7a0b0d2b8e484b8b515059bdb0cf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:11.863095 11837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7a0b0d2b8e484b8b515059bdb0cf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:11.865017 11983 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7a0b0d2b8e484b8b515059bdb0cf67 (DEFAULT_TABLE table=test-table [id=dae21e5ff7084ea4945078bd87354553]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:11.866050 11983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7a0b0d2b8e484b8b515059bdb0cf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:11.877955 12226 tablet_bootstrap.cc:492] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: Bootstrap starting.
I20260620 20:10:11.880833 12226 tablet_bootstrap.cc:654] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:11.881693 12228 tablet_bootstrap.cc:492] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: Bootstrap starting.
I20260620 20:10:11.882477 12227 tablet_bootstrap.cc:492] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: Bootstrap starting.
I20260620 20:10:11.883280 12226 log.cc:824] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:11.885979 12228 tablet_bootstrap.cc:654] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:11.886282 12227 tablet_bootstrap.cc:654] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:11.886538 12226 tablet_bootstrap.cc:492] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: No bootstrap required, opened a new log
I20260620 20:10:11.886801 12226 ts_tablet_manager.cc:1397] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:10:11.887063 12228 log.cc:824] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:11.887454 12227 log.cc:824] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:11.890070 12228 tablet_bootstrap.cc:492] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: No bootstrap required, opened a new log
I20260620 20:10:11.890327 12228 ts_tablet_manager.cc:1397] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:10:11.890637 12227 tablet_bootstrap.cc:492] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: No bootstrap required, opened a new log
I20260620 20:10:11.890921 12227 ts_tablet_manager.cc:1397] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260620 20:10:11.897831 12226 raft_consensus.cc:348] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } }
I20260620 20:10:11.898273 12226 raft_consensus.cc:374] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:11.898366 12226 raft_consensus.cc:729] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f359aa5fee0d4e49949dc64a62e96dbb, State: Initialized, Role: FOLLOWER
I20260620 20:10:11.898363 12227 raft_consensus.cc:348] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } }
I20260620 20:10:11.898823 12227 raft_consensus.cc:374] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:11.898937 12227 raft_consensus.cc:729] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e85269446e48feb97c9899cc44b756, State: Initialized, Role: FOLLOWER
I20260620 20:10:11.898995 12226 consensus_queue.cc:260] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } }
I20260620 20:10:11.899555 12227 consensus_queue.cc:260] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } }
I20260620 20:10:11.900728 12226 ts_tablet_manager.cc:1428] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 20:10:11.901001 12228 raft_consensus.cc:348] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } }
I20260620 20:10:11.901407 12228 raft_consensus.cc:374] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:11.901502 12228 raft_consensus.cc:729] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d7e639f3ee49efab35348707f2a64e, State: Initialized, Role: FOLLOWER
I20260620 20:10:11.902048 12228 consensus_queue.cc:260] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } }
I20260620 20:10:11.902627 12049 heartbeater.cc:503] Master 127.4.169.190:38587 was elected leader, sending a full tablet report...
I20260620 20:10:11.918727 12228 ts_tablet_manager.cc:1428] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: Time spent starting tablet: real 0.028s user 0.019s sys 0.008s
I20260620 20:10:11.919337 12227 ts_tablet_manager.cc:1428] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: Time spent starting tablet: real 0.028s user 0.010s sys 0.006s
I20260620 20:10:11.920076 12190 heartbeater.cc:503] Master 127.4.169.190:38587 was elected leader, sending a full tablet report...
W20260620 20:10:11.964897 11905 tablet.cc:2369] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:10:11.970168 12192 tablet.cc:2369] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:10:11.980512 12233 raft_consensus.cc:482] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:10:11.980818 12233 raft_consensus.cc:504] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } }
I20260620 20:10:11.982522 12233 leader_election.cc:288] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23d7e639f3ee49efab35348707f2a64e (127.4.169.131:40573), f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699)
I20260620 20:10:11.991298 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.982933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:43281 (local address 127.4.169.131:40573)
0620 20:10:11.983182 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.983191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.983975 (+ 784us) server_negotiation.cc:407] Connection header received
0620 20:10:11.984290 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.984298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.984394 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.984504 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:11.985229 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.986583 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.989414 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.990363 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.990593 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.990669 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.990933 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.991052 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:10:11.991420 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.982666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:40573 (local address 127.4.169.129:43281)
0620 20:10:11.983839 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.983864 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.984106 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.984698 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.984704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.984731 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:11.985086 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.985100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.986761 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.986769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.989274 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.989284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.990614 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.990620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.990770 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.990794 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.990908 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.991037 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":770,"threads_started":1}
I20260620 20:10:11.992458 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.983581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:47579 (local address 127.4.169.130:41699)
0620 20:10:11.983813 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:11.983821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:11.984149 (+ 328us) server_negotiation.cc:407] Connection header received
0620 20:10:11.984382 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:11.984386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:11.984440 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:11.984535 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:11.985267 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.985980 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.989153 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.990778 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.990973 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.991074 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:11.992068 (+ 994us) server_negotiation.cc:299] Negotiation successful
0620 20:10:11.992205 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:10:11.992457 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7a0b0d2b8e484b8b515059bdb0cf67" candidate_uuid: "c3e85269446e48feb97c9899cc44b756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d7e639f3ee49efab35348707f2a64e" is_pre_election: true
I20260620 20:10:11.992935 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:11.983040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:41699 (local address 127.4.169.129:47579)
0620 20:10:11.984060 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:11.984075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:11.984236 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:11.984552 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:11.984556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:11.984580 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:11.985048 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.985065 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.986114 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.986119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.988982 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:11.988996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:11.991276 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:11.991285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:11.991861 (+ 576us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:11.991916 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 20:10:11.992033 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 20:10:11.992221 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":4,"thread_start_us":858,"threads_started":1}
I20260620 20:10:11.993113 12144 raft_consensus.cc:2454] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e85269446e48feb97c9899cc44b756 in term 0.
I20260620 20:10:11.993966 11790 leader_election.cc:302] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d7e639f3ee49efab35348707f2a64e, c3e85269446e48feb97c9899cc44b756; no voters:
I20260620 20:10:11.993922 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7a0b0d2b8e484b8b515059bdb0cf67" candidate_uuid: "c3e85269446e48feb97c9899cc44b756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" is_pre_election: true
I20260620 20:10:11.994542 12233 raft_consensus.cc:2781] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:10:11.994534 12003 raft_consensus.cc:2454] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3e85269446e48feb97c9899cc44b756 in term 0.
I20260620 20:10:11.994711 12233 raft_consensus.cc:482] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:10:11.994828 12233 raft_consensus.cc:3037] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:11.998773 12233 raft_consensus.cc:504] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } }
I20260620 20:10:11.999805 12233 leader_election.cc:288] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [CANDIDATE]: Term 1 election: Requested vote from peers 23d7e639f3ee49efab35348707f2a64e (127.4.169.131:40573), f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699)
I20260620 20:10:12.000016 12144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7a0b0d2b8e484b8b515059bdb0cf67" candidate_uuid: "c3e85269446e48feb97c9899cc44b756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d7e639f3ee49efab35348707f2a64e"
I20260620 20:10:12.000057 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7a0b0d2b8e484b8b515059bdb0cf67" candidate_uuid: "c3e85269446e48feb97c9899cc44b756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f359aa5fee0d4e49949dc64a62e96dbb"
I20260620 20:10:12.000234 12003 raft_consensus.cc:3037] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:12.000234 12144 raft_consensus.cc:3037] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:12.003456 12003 raft_consensus.cc:2454] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e85269446e48feb97c9899cc44b756 in term 1.
I20260620 20:10:12.003456 12144 raft_consensus.cc:2454] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3e85269446e48feb97c9899cc44b756 in term 1.
I20260620 20:10:12.003886 11792 leader_election.cc:302] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3e85269446e48feb97c9899cc44b756, f359aa5fee0d4e49949dc64a62e96dbb; no voters:
I20260620 20:10:12.004307 12233 raft_consensus.cc:2781] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:12.004810 12233 raft_consensus.cc:686] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 1 LEADER]: Becoming Leader. State: Replica: c3e85269446e48feb97c9899cc44b756, State: Running, Role: LEADER
I20260620 20:10:12.005872 12233 consensus_queue.cc:237] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } }
I20260620 20:10:12.015882 10933 catalog_manager.cc:5495] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3e85269446e48feb97c9899cc44b756 (127.4.169.129). New cstate: current_term: 1 leader_uuid: "c3e85269446e48feb97c9899cc44b756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } health_report { overall_health: HEALTHY } } }
I20260620 20:10:12.057651 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:12.051547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.4.169.129:35319)
0620 20:10:12.051759 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:12.051767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:12.052009 (+ 242us) server_negotiation.cc:407] Connection header received
0620 20:10:12.052278 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:12.052290 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:12.052377 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:12.052532 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:10:12.052898 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.053650 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.055084 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.055938 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.056085 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:12.056562 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:10:12.057079 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:10:12.057240 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:12.057304 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 20:10:12.057383 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:10:12.066047 12144 raft_consensus.cc:1261] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [term 1 FOLLOWER]: Refusing update from remote peer c3e85269446e48feb97c9899cc44b756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:12.066090 12003 raft_consensus.cc:1261] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [term 1 FOLLOWER]: Refusing update from remote peer c3e85269446e48feb97c9899cc44b756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:12.067134 12237 consensus_queue.cc:1035] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:12.068061 12237 consensus_queue.cc:1035] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:12.129235 10933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:12.129503 10933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:12.138638 10933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7c832b7122d493794c7c2731fa72596: Sending DeleteTablet for 3 replicas of tablet 8a7a0b0d2b8e484b8b515059bdb0cf67
I20260620 20:10:12.139670 11983 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7a0b0d2b8e484b8b515059bdb0cf67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:12 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:53412
I20260620 20:10:12.139720 12124 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7a0b0d2b8e484b8b515059bdb0cf67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:12 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:43964
I20260620 20:10:12.140988 11837 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7a0b0d2b8e484b8b515059bdb0cf67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:12 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:57034
I20260620 20:10:12.141355 12250 tablet_replica.cc:330] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: stopping tablet replica
I20260620 20:10:12.141848 12250 raft_consensus.cc:2229] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:12.142431 12250 raft_consensus.cc:2258] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:12.150434 12252 tablet_replica.cc:330] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: stopping tablet replica
I20260620 20:10:12.151183 12252 raft_consensus.cc:2229] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:10:12.152122 12252 raft_consensus.cc:2258] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:12.153025 12250 ts_tablet_manager.cc:1905] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:12.160080 12252 ts_tablet_manager.cc:1905] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:12.169094 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 10901
I20260620 20:10:12.169895 12251 tablet_replica.cc:330] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: stopping tablet replica
I20260620 20:10:12.170259 10963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:12.170941 12251 raft_consensus.cc:2229] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:12.173784 12251 raft_consensus.cc:2258] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:12.176452 12251 ts_tablet_manager.cc:1905] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:12.196094 12250 ts_tablet_manager.cc:1918] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:12.198494 12252 ts_tablet_manager.cc:1918] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:12.201387 12250 log.cc:1196] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/8a7a0b0d2b8e484b8b515059bdb0cf67
I20260620 20:10:12.202131 12250 ts_tablet_manager.cc:1939] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P f359aa5fee0d4e49949dc64a62e96dbb: Deleting consensus metadata
I20260620 20:10:12.204108 12252 log.cc:1196] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/8a7a0b0d2b8e484b8b515059bdb0cf67
I20260620 20:10:12.204857 12252 ts_tablet_manager.cc:1939] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P c3e85269446e48feb97c9899cc44b756: Deleting consensus metadata
I20260620 20:10:12.214808 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:12.142470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.4.169.189:36339)
0620 20:10:12.142688 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:12.142695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:12.167867 (+ 25172us) server_negotiation.cc:407] Connection header received
0620 20:10:12.167978 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:12.167985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:12.168088 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:12.168230 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:12.176934 (+ 8704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.178447 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.191579 (+ 13132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.192490 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.192620 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:12.205142 (+ 12522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:12.205150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:12.205156 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:12.205212 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:12.209348 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:12.212202 (+ 2854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:12.212210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:12.212215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:12.212397 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:12.212873 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:12.212876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:12.212879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:12.213279 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:12.213545 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:12.214222 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 20:10:12.214491 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:10:12.219664 12251 ts_tablet_manager.cc:1918] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:12.219957 12251 log.cc:1196] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/8a7a0b0d2b8e484b8b515059bdb0cf67
I20260620 20:10:12.220552 12251 ts_tablet_manager.cc:1939] T 8a7a0b0d2b8e484b8b515059bdb0cf67 P 23d7e639f3ee49efab35348707f2a64e: Deleting consensus metadata
I20260620 20:10:12.237800 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:12.167919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.4.169.188:40759)
0620 20:10:12.168113 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:12.168121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:12.217357 (+ 49236us) server_negotiation.cc:407] Connection header received
0620 20:10:12.217506 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:12.217511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:12.217591 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:12.217681 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:12.223960 (+ 6279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.225089 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.227063 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.227757 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.227853 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:12.231378 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:12.231385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:12.231390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:12.231431 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:12.234875 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:12.235666 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:12.235672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:12.235676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:12.235805 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:12.236203 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:12.236208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:12.236212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:12.236535 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:12.236760 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:12.237249 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 20:10:12.237462 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:10:12.336426 10917 catalog_manager.cc:4841] TS 23d7e639f3ee49efab35348707f2a64e (127.4.169.131:40573): tablet 8a7a0b0d2b8e484b8b515059bdb0cf67 (table test-table [id=dae21e5ff7084ea4945078bd87354553]) successfully deleted
I20260620 20:10:12.337703 10918 catalog_manager.cc:4841] TS c3e85269446e48feb97c9899cc44b756 (127.4.169.129:35319): tablet 8a7a0b0d2b8e484b8b515059bdb0cf67 (table test-table [id=dae21e5ff7084ea4945078bd87354553]) successfully deleted
I20260620 20:10:12.337826 10919 catalog_manager.cc:4841] TS f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699): tablet 8a7a0b0d2b8e484b8b515059bdb0cf67 (table test-table [id=dae21e5ff7084ea4945078bd87354553]) successfully deleted
I20260620 20:10:12.338511 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 10901
I20260620 20:10:12.383378 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:12.365072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.4.169.188:40759)
0620 20:10:12.365318 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:12.365327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:12.366242 (+ 915us) server_negotiation.cc:407] Connection header received
0620 20:10:12.366541 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:12.366547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:12.366627 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:12.366709 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:12.369772 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.370968 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.372806 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.373206 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.373363 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:12.376939 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:12.376945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:12.376950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:12.376991 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:12.380121 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:12.380885 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:12.380891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:12.380895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:12.381034 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:12.381507 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:12.381514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:12.381520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:12.381898 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:12.382156 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:12.382615 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 20:10:12.383029 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 20:10:12.385492 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:12.364156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.4.169.189:36339)
0620 20:10:12.364403 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:12.364412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:12.365997 (+ 1585us) server_negotiation.cc:407] Connection header received
0620 20:10:12.366361 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:12.366370 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:12.366504 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:12.366596 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:12.369982 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.371075 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.374671 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:12.375377 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:12.375531 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:12.378663 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:12.378669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:12.378674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:12.378725 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:12.382063 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:12.383147 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:12.383154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:12.383157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:12.383300 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:12.384052 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:12.384057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:12.384061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:12.384386 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:12.384616 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:12.384953 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 20:10:12.385172 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:10:13.109344 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.108452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:36781)
0620 20:10:13.108967 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.109081 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":281,"threads_started":1}
W20260620 20:10:13.109894 12190 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:38587 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:13.116420 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.115617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.130:47377)
0620 20:10:13.116121 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.116276 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":281,"threads_started":1}
W20260620 20:10:13.116796 12049 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:38587 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:13.127259 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.125113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.129:33849)
0620 20:10:13.126865 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.127023 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1649,"thread_start_us":1563,"threads_started":1}
W20260620 20:10:13.127656 11903 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:38587 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:13.637812 12265 raft_consensus.cc:482] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c832b7122d493794c7c2731fa72596)
I20260620 20:10:13.638119 12265 raft_consensus.cc:504] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } }
I20260620 20:10:13.638249 12266 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c832b7122d493794c7c2731fa72596)
I20260620 20:10:13.638576 12266 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } }
I20260620 20:10:13.639794 12266 leader_election.cc:288] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587), f55538c2647943c1b75637fdb733692b (127.4.169.188:40759)
I20260620 20:10:13.639577 12265 leader_election.cc:288] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587), dc5844a44fc04b1eb9479d71b25ace63 (127.4.169.189:36339)
I20260620 20:10:13.640966 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.639752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55280)
0620 20:10:13.640474 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.640686 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":341,"threads_started":1}
W20260620 20:10:13.641214 11707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:13.641969 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.639902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55288)
0620 20:10:13.641277 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.641583 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1250,"thread_start_us":1039,"threads_started":1}
I20260620 20:10:13.644547 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.644074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55298)
0620 20:10:13.644313 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.644396 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260620 20:10:13.644454 11517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
W20260620 20:10:13.644694 11707 leader_election.cc:334] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587): Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:13.646765 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.646119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55310)
0620 20:10:13.646347 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.646540 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 20:10:13.647014 11517 leader_election.cc:334] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587): Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:13.651556 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.641292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:40759 (local address 127.0.0.1:59186)
0620 20:10:13.642492 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.642516 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:13.642663 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:13.644784 (+ 2121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:13.644792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:13.644829 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:13.645259 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:13.645274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:13.646674 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:13.646689 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:13.648805 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:13.648819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:13.650375 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:13.650418 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:13.650729 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:13.650841 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 20:10:13.650962 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 20:10:13.651165 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":72,"thread_start_us":2416,"threads_started":1}
I20260620 20:10:13.651574 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.641540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.4.169.188:40759)
0620 20:10:13.644343 (+ 2803us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:13.644348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:13.644371 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:10:13.644437 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:13.644444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:13.644520 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:13.645413 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:13.645583 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:13.646544 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:13.648942 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:13.650246 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:13.650370 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:13.650520 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:13.651007 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 20:10:13.651147 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2671,"thread_start_us":2514,"threads_started":1}
I20260620 20:10:13.651643 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.639784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:36339 (local address 127.0.0.1:36628)
0620 20:10:13.640819 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.640838 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:13.640983 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:13.644829 (+ 3846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:13.644849 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:13.644873 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:13.645214 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:13.645227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:13.647063 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:13.647078 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:13.649028 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:13.649039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:13.650688 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:13.650696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:13.650894 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:13.650940 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 20:10:13.651042 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:10:13.651194 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":3,"thread_start_us":755,"threads_started":1}
I20260620 20:10:13.651573 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.643902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.4.169.189:36339)
0620 20:10:13.644488 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:13.644494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:13.644511 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:10:13.644578 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:13.644584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:13.644655 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:13.645372 (+ 717us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:13.645636 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:13.646922 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:13.649154 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:13.650489 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:13.650653 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:13.650909 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:13.651060 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 20:10:13.651180 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":470,"thread_start_us":293,"threads_started":1}
I20260620 20:10:13.652287 11733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc5844a44fc04b1eb9479d71b25ace63" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f55538c2647943c1b75637fdb733692b" is_pre_election: true
I20260620 20:10:13.652598 11733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc5844a44fc04b1eb9479d71b25ace63 in term 1.
I20260620 20:10:13.653040 11543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f55538c2647943c1b75637fdb733692b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc5844a44fc04b1eb9479d71b25ace63" is_pre_election: true
I20260620 20:10:13.653087 11517 leader_election.cc:302] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc5844a44fc04b1eb9479d71b25ace63, f55538c2647943c1b75637fdb733692b; no voters: e7c832b7122d493794c7c2731fa72596
I20260620 20:10:13.653352 11543 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f55538c2647943c1b75637fdb733692b in term 1.
I20260620 20:10:13.653504 12266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 20:10:13.653640 12266 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e7c832b7122d493794c7c2731fa72596)
I20260620 20:10:13.653740 12266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:10:13.653889 11710 leader_election.cc:302] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc5844a44fc04b1eb9479d71b25ace63, f55538c2647943c1b75637fdb733692b; no voters: e7c832b7122d493794c7c2731fa72596
I20260620 20:10:13.654176 12265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 20:10:13.654290 12265 raft_consensus.cc:482] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Starting leader election (detected failure of leader e7c832b7122d493794c7c2731fa72596)
I20260620 20:10:13.654378 12265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:10:13.655807 12266 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } }
I20260620 20:10:13.656358 12266 leader_election.cc:288] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 2 election: Requested vote from peers e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587), f55538c2647943c1b75637fdb733692b (127.4.169.188:40759)
I20260620 20:10:13.656471 12265 raft_consensus.cc:504] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } }
I20260620 20:10:13.656942 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.656524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55312)
0620 20:10:13.656658 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.656779 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 20:10:13.657035 11733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc5844a44fc04b1eb9479d71b25ace63" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f55538c2647943c1b75637fdb733692b"
I20260620 20:10:13.657186 12265 leader_election.cc:288] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [CANDIDATE]: Term 2 election: Requested vote from peers e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587), dc5844a44fc04b1eb9479d71b25ace63 (127.4.169.189:36339)
I20260620 20:10:13.657383 11733 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dc5844a44fc04b1eb9479d71b25ace63 in current term 2: Already voted for candidate f55538c2647943c1b75637fdb733692b in this term.
I20260620 20:10:13.657620 11543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f55538c2647943c1b75637fdb733692b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc5844a44fc04b1eb9479d71b25ace63"
I20260620 20:10:13.657869 11543 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f55538c2647943c1b75637fdb733692b in current term 2: Already voted for candidate dc5844a44fc04b1eb9479d71b25ace63 in this term.
I20260620 20:10:13.657935 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.657498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55318)
0620 20:10:13.657697 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.657795 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 20:10:13.659166 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.658581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55334)
0620 20:10:13.658913 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.659006 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
W20260620 20:10:13.659313 11707 leader_election.cc:334] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587): Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:13.659435 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:13.659033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55336)
0620 20:10:13.659198 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:13.659291 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 20:10:13.659432 11707 leader_election.cc:302] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f55538c2647943c1b75637fdb733692b; no voters: dc5844a44fc04b1eb9479d71b25ace63, e7c832b7122d493794c7c2731fa72596
W20260620 20:10:13.659603 11517 leader_election.cc:334] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587): Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:13.659785 11517 leader_election.cc:302] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc5844a44fc04b1eb9479d71b25ace63; no voters: e7c832b7122d493794c7c2731fa72596, f55538c2647943c1b75637fdb733692b
I20260620 20:10:13.659847 12265 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 20:10:13.660048 12266 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 20:10:14.111544 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:14.110689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:35859)
0620 20:10:14.111190 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:14.111327 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":285,"threads_started":1}
I20260620 20:10:14.118258 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:14.117448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.130:50953)
0620 20:10:14.117973 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:14.118074 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":293,"threads_started":1}
I20260620 20:10:14.129046 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:14.128314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.129:44939)
0620 20:10:14.128745 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:14.128863 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":211,"threads_started":1}
I20260620 20:10:15.113938 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.112392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:52277)
0620 20:10:15.113614 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.113722 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1082,"thread_start_us":969,"threads_started":1}
I20260620 20:10:15.119828 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.119014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.130:56427)
0620 20:10:15.119558 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.119670 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":302,"threads_started":1}
I20260620 20:10:15.130698 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.129922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.129:39853)
0620 20:10:15.130459 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.130551 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":289,"threads_started":1}
I20260620 20:10:15.297137 12281 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:10:15.297324 12281 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } }
I20260620 20:10:15.297916 12281 leader_election.cc:288] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587), f55538c2647943c1b75637fdb733692b (127.4.169.188:40759)
I20260620 20:10:15.298919 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.298086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55598)
0620 20:10:15.298571 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.298698 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":194,"threads_started":1}
I20260620 20:10:15.299013 11733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc5844a44fc04b1eb9479d71b25ace63" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f55538c2647943c1b75637fdb733692b" is_pre_election: true
I20260620 20:10:15.299278 11733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc5844a44fc04b1eb9479d71b25ace63 in term 2.
I20260620 20:10:15.299700 11517 leader_election.cc:302] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc5844a44fc04b1eb9479d71b25ace63, f55538c2647943c1b75637fdb733692b; no voters:
I20260620 20:10:15.299968 12281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 20:10:15.300069 12281 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:10:15.300190 12281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:10:15.300462 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.300101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55600)
0620 20:10:15.300270 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.300356 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 20:10:15.300637 11517 leader_election.cc:334] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587): Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:15.302600 12281 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } }
I20260620 20:10:15.303112 12281 leader_election.cc:288] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 3 election: Requested vote from peers e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587), f55538c2647943c1b75637fdb733692b (127.4.169.188:40759)
I20260620 20:10:15.303551 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.303187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55614)
0620 20:10:15.303335 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.303447 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:10:15.303699 11733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc5844a44fc04b1eb9479d71b25ace63" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f55538c2647943c1b75637fdb733692b"
I20260620 20:10:15.303949 11733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:10:15.304596 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.304267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55616)
0620 20:10:15.304418 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.304489 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:10:15.304739 11517 leader_election.cc:334] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587): Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
I20260620 20:10:15.306058 11733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc5844a44fc04b1eb9479d71b25ace63 in term 3.
I20260620 20:10:15.306521 11517 leader_election.cc:302] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc5844a44fc04b1eb9479d71b25ace63, f55538c2647943c1b75637fdb733692b; no voters: e7c832b7122d493794c7c2731fa72596
I20260620 20:10:15.306785 12281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:10:15.307199 12281 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [term 3 LEADER]: Becoming Leader. State: Replica: dc5844a44fc04b1eb9479d71b25ace63, State: Running, Role: LEADER
I20260620 20:10:15.307512 12281 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } }
I20260620 20:10:15.309105 12284 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc5844a44fc04b1eb9479d71b25ace63. Latest consensus state: current_term: 3 leader_uuid: "dc5844a44fc04b1eb9479d71b25ace63" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:15.309448 12284 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:15.310051 12286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:15.313475 12286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:15.314268 12286 catalog_manager.cc:1259] Loaded cluster ID: db37cee9ee884b908d24568ad4881b07
I20260620 20:10:15.314368 12286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:15.315613 12286 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:15.316512 12286 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Loaded TSK: 0
I20260620 20:10:15.317409 12286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:10:15.344504 11533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:10:15.351459 11533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:10:15.355463 11733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [term 3 FOLLOWER]: Refusing update from remote peer dc5844a44fc04b1eb9479d71b25ace63: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 20:10:15.355496 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.355119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55626)
0620 20:10:15.355297 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.355365 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 20:10:15.356156 12284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 20:10:15.356667 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.356275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55632)
0620 20:10:15.356471 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.356541 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:10:15.356871 11517 consensus_peers.cc:487] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 -> Peer e7c832b7122d493794c7c2731fa72596 (127.4.169.190:38587): Couldn't send request to peer e7c832b7122d493794c7c2731fa72596. Status: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 20:10:15.358971 12288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc5844a44fc04b1eb9479d71b25ace63. Latest consensus state: current_term: 3 leader_uuid: "dc5844a44fc04b1eb9479d71b25ace63" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:15.359272 12288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:10:15.360512 12284 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "dc5844a44fc04b1eb9479d71b25ace63" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:15.360826 12284 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:15.361125 12288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "dc5844a44fc04b1eb9479d71b25ace63" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e7c832b7122d493794c7c2731fa72596" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38587 } } peers { permanent_uuid: "dc5844a44fc04b1eb9479d71b25ace63" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 36339 } attrs { promote: false } } peers { permanent_uuid: "f55538c2647943c1b75637fdb733692b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 40759 } attrs { promote: false } } }
I20260620 20:10:15.361418 12288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55538c2647943c1b75637fdb733692b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:10:15.380297 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.368620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:41699 (local address 127.0.0.1:53422)
0620 20:10:15.368792 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.368812 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:15.368941 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:15.371236 (+ 2295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:15.371244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:15.371281 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:15.371701 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.371716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.374312 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.374321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.377011 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.377032 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.379203 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.379212 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.379519 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.379587 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 20:10:15.379672 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:10:15.379866 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 20:10:15.382705 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.368747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.4.169.130:41699)
0620 20:10:15.370843 (+ 2096us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:15.370851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:15.370870 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:10:15.370940 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:15.370949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:15.371041 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:15.371845 (+ 804us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:15.372036 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.374125 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.377267 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.378953 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.382082 (+ 3129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.382182 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:15.382229 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 20:10:15.382350 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":1788,"threads_started":1}
I20260620 20:10:15.383426 11983 tablet_service.cc:1467] Processing CreateTablet for tablet 6634631c6cfb4efeb8947f3263a8cf1d (DEFAULT_TABLE table=test-table [id=6e985c01771d460b899b83dceac088b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:15.383915 11983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6634631c6cfb4efeb8947f3263a8cf1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:15.388363 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.369090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:35319 (local address 127.0.0.1:57054)
0620 20:10:15.369818 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.369832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:15.369915 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:15.381319 (+ 11404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:15.381327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:15.381361 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:15.381866 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.381876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.383431 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.383436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.385436 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.385450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.387375 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.387381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.387557 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.387598 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:10:15.387692 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 20:10:15.387868 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":3,"thread_start_us":535,"threads_started":1}
I20260620 20:10:15.388513 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.369489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.4.169.129:35319)
0620 20:10:15.380887 (+ 11398us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:15.380897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:15.380923 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:10:15.381008 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:15.381016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:15.381119 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:15.381240 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:15.382019 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.383271 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.385603 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.387190 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.387323 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.387435 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:15.388058 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 20:10:15.388203 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11230,"thread_start_us":11923,"threads_started":1}
I20260620 20:10:15.389478 11837 tablet_service.cc:1467] Processing CreateTablet for tablet 6634631c6cfb4efeb8947f3263a8cf1d (DEFAULT_TABLE table=test-table [id=6e985c01771d460b899b83dceac088b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:15.389911 11837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6634631c6cfb4efeb8947f3263a8cf1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:15.390441 12298 tablet_bootstrap.cc:492] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: Bootstrap starting.
I20260620 20:10:15.392486 12298 tablet_bootstrap.cc:654] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:15.393491 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.370881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:40573 (local address 127.0.0.1:43976)
0620 20:10:15.380592 (+ 9711us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.380615 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:15.380749 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:15.387653 (+ 6904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:15.387657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:15.387681 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:15.387877 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.387885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.389152 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.389159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.390779 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.390793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.392612 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.392618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.392841 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.392877 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:10:15.392975 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:10:15.393134 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9569,"mutex_wait_us":32,"thread_start_us":10842,"threads_started":1}
I20260620 20:10:15.393548 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.371056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.4.169.131:40573)
0620 20:10:15.387203 (+ 16147us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:15.387211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:15.387234 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:10:15.387313 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:15.387318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:15.387403 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:15.387637 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:15.388011 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.389034 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.390967 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.392266 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.392402 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.392523 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:15.393022 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 20:10:15.393220 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15985,"thread_start_us":16416,"threads_started":1}
I20260620 20:10:15.394788 12124 tablet_service.cc:1467] Processing CreateTablet for tablet 6634631c6cfb4efeb8947f3263a8cf1d (DEFAULT_TABLE table=test-table [id=6e985c01771d460b899b83dceac088b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:15.395207 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6634631c6cfb4efeb8947f3263a8cf1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:15.396559 12300 tablet_bootstrap.cc:492] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: Bootstrap starting.
I20260620 20:10:15.398941 12300 tablet_bootstrap.cc:654] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:15.401000 12301 tablet_bootstrap.cc:492] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: Bootstrap starting.
I20260620 20:10:15.401144 12298 tablet_bootstrap.cc:492] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: No bootstrap required, opened a new log
I20260620 20:10:15.401414 12298 ts_tablet_manager.cc:1397] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260620 20:10:15.402540 12298 raft_consensus.cc:348] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } }
I20260620 20:10:15.402803 12298 raft_consensus.cc:374] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:15.402856 12301 tablet_bootstrap.cc:654] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:15.402909 12298 raft_consensus.cc:729] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f359aa5fee0d4e49949dc64a62e96dbb, State: Initialized, Role: FOLLOWER
I20260620 20:10:15.403208 12298 consensus_queue.cc:260] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } }
I20260620 20:10:15.403723 12300 tablet_bootstrap.cc:492] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: No bootstrap required, opened a new log
I20260620 20:10:15.403918 12300 ts_tablet_manager.cc:1397] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20260620 20:10:15.404948 12300 raft_consensus.cc:348] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } }
I20260620 20:10:15.405333 12300 raft_consensus.cc:374] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:15.405503 12300 raft_consensus.cc:729] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e85269446e48feb97c9899cc44b756, State: Initialized, Role: FOLLOWER
I20260620 20:10:15.405897 12300 consensus_queue.cc:260] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } }
I20260620 20:10:15.410234 12300 ts_tablet_manager.cc:1428] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260620 20:10:15.411448 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.411026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.129:47237)
0620 20:10:15.411193 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.411305 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:10:15.412566 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.412104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.129:50559)
0620 20:10:15.412335 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.412420 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 20:10:15.414193 11904 heartbeater.cc:503] Master 127.4.169.189:36339 was elected leader, sending a full tablet report...
I20260620 20:10:15.415084 12298 ts_tablet_manager.cc:1428] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: Time spent starting tablet: real 0.014s user 0.007s sys 0.003s
I20260620 20:10:15.416788 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.415752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.130:39931)
0620 20:10:15.415966 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.416051 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 20:10:15.417351 12050 heartbeater.cc:503] Master 127.4.169.189:36339 was elected leader, sending a full tablet report...
I20260620 20:10:15.418089 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.417560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.130:39975)
0620 20:10:15.417840 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.417946 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260620 20:10:15.418112 12301 tablet_bootstrap.cc:492] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: No bootstrap required, opened a new log
I20260620 20:10:15.418286 12301 ts_tablet_manager.cc:1397] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260620 20:10:15.419179 12301 raft_consensus.cc:348] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } }
I20260620 20:10:15.419440 12301 raft_consensus.cc:374] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:15.419579 12301 raft_consensus.cc:729] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d7e639f3ee49efab35348707f2a64e, State: Initialized, Role: FOLLOWER
I20260620 20:10:15.419838 12301 consensus_queue.cc:260] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } }
I20260620 20:10:15.420665 12301 ts_tablet_manager.cc:1428] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:10:15.421677 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.421125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:37859)
0620 20:10:15.421389 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.421492 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 20:10:15.422570 12191 heartbeater.cc:503] Master 127.4.169.189:36339 was elected leader, sending a full tablet report...
I20260620 20:10:15.422786 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.422315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:32925)
0620 20:10:15.422478 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.422571 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 20:10:15.607594 12306 raft_consensus.cc:482] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:10:15.607841 12306 raft_consensus.cc:504] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } }
I20260620 20:10:15.609448 12306 leader_election.cc:288] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699), c3e85269446e48feb97c9899cc44b756 (127.4.169.129:35319)
W20260620 20:10:15.611475 12051 tablet.cc:2369] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:10:15.616691 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.609809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:44749 (local address 127.4.169.130:41699)
0620 20:10:15.610141 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:15.610151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:15.610180 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:10:15.610440 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:15.610446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:15.610525 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:15.610617 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:15.611159 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.612181 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.614547 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.615757 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.615910 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.616020 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:15.616301 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 20:10:15.616436 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 20:10:15.616775 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.609869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:58117 (local address 127.4.169.129:35319)
0620 20:10:15.610151 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:15.610157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:15.610774 (+ 617us) server_negotiation.cc:407] Connection header received
0620 20:10:15.610917 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:15.610921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:15.610974 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:15.611042 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:15.611640 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.612524 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.614717 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.615916 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.616068 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.616123 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:15.616404 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 20:10:15.616522 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 20:10:15.616802 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.609777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:41699 (local address 127.4.169.131:44749)
0620 20:10:15.610064 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.610083 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:15.610262 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:15.610668 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:15.610673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:15.610708 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:15.610993 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.611002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.612316 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.612320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.614406 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.614423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.615923 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.615927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.616137 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.616180 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 20:10:15.616282 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:10:15.616427 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 20:10:15.616937 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.609821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:35319 (local address 127.4.169.131:58117)
0620 20:10:15.610653 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.610670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:15.610794 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:15.611092 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:15.611097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:15.611124 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:15.611450 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.611465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.612642 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.612646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.614566 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:15.614584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.616087 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.616092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:15.616260 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.616295 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:10:15.616368 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:10:15.616470 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":1,"thread_start_us":470,"threads_started":1}
I20260620 20:10:15.617792 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6634631c6cfb4efeb8947f3263a8cf1d" candidate_uuid: "23d7e639f3ee49efab35348707f2a64e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" is_pre_election: true
I20260620 20:10:15.617825 11857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6634631c6cfb4efeb8947f3263a8cf1d" candidate_uuid: "23d7e639f3ee49efab35348707f2a64e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e85269446e48feb97c9899cc44b756" is_pre_election: true
I20260620 20:10:15.618026 12003 raft_consensus.cc:2454] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d7e639f3ee49efab35348707f2a64e in term 0.
I20260620 20:10:15.618249 11857 raft_consensus.cc:2454] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d7e639f3ee49efab35348707f2a64e in term 0.
I20260620 20:10:15.618643 12079 leader_election.cc:302] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d7e639f3ee49efab35348707f2a64e, f359aa5fee0d4e49949dc64a62e96dbb; no voters:
I20260620 20:10:15.619381 12306 raft_consensus.cc:2781] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:10:15.619525 12306 raft_consensus.cc:482] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:10:15.619626 12306 raft_consensus.cc:3037] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:15.622030 12306 raft_consensus.cc:504] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } }
I20260620 20:10:15.622700 12306 leader_election.cc:288] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [CANDIDATE]: Term 1 election: Requested vote from peers f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699), c3e85269446e48feb97c9899cc44b756 (127.4.169.129:35319)
I20260620 20:10:15.623042 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6634631c6cfb4efeb8947f3263a8cf1d" candidate_uuid: "23d7e639f3ee49efab35348707f2a64e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f359aa5fee0d4e49949dc64a62e96dbb"
I20260620 20:10:15.623229 12003 raft_consensus.cc:3037] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:15.623206 11857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6634631c6cfb4efeb8947f3263a8cf1d" candidate_uuid: "23d7e639f3ee49efab35348707f2a64e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e85269446e48feb97c9899cc44b756"
I20260620 20:10:15.623399 11857 raft_consensus.cc:3037] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:15.625398 11857 raft_consensus.cc:2454] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d7e639f3ee49efab35348707f2a64e in term 1.
I20260620 20:10:15.625437 12003 raft_consensus.cc:2454] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d7e639f3ee49efab35348707f2a64e in term 1.
I20260620 20:10:15.625900 12078 leader_election.cc:302] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d7e639f3ee49efab35348707f2a64e, c3e85269446e48feb97c9899cc44b756; no voters:
I20260620 20:10:15.626200 12306 raft_consensus.cc:2781] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:15.626711 12306 raft_consensus.cc:686] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 1 LEADER]: Becoming Leader. State: Replica: 23d7e639f3ee49efab35348707f2a64e, State: Running, Role: LEADER
I20260620 20:10:15.627115 12306 consensus_queue.cc:237] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } }
I20260620 20:10:15.627980 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.627535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:51317)
0620 20:10:15.627698 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.627815 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 20:10:15.630960 11532 catalog_manager.cc:5495] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e reported cstate change: term changed from 0 to 1, leader changed from <none> to 23d7e639f3ee49efab35348707f2a64e (127.4.169.131). New cstate: current_term: 1 leader_uuid: "23d7e639f3ee49efab35348707f2a64e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23d7e639f3ee49efab35348707f2a64e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40573 } health_report { overall_health: HEALTHY } } }
I20260620 20:10:15.695245 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.689967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.4.169.131:40573)
0620 20:10:15.690155 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:15.690160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:15.690177 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:10:15.690362 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:15.690366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:15.690524 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:15.690646 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:10:15.690942 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.691609 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.692862 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.693405 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.693498 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.693887 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:10:15.694622 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:10:15.694735 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:15.694882 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 20:10:15.694986 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:10:15.699112 12003 raft_consensus.cc:1261] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [term 1 FOLLOWER]: Refusing update from remote peer 23d7e639f3ee49efab35348707f2a64e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:15.699448 11857 raft_consensus.cc:1261] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [term 1 FOLLOWER]: Refusing update from remote peer 23d7e639f3ee49efab35348707f2a64e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:15.699851 12308 consensus_queue.cc:1035] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f359aa5fee0d4e49949dc64a62e96dbb" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:15.700184 12306 consensus_queue.cc:1035] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e85269446e48feb97c9899cc44b756" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:15.702754 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.702105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.130:60481)
0620 20:10:15.702302 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.702458 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 20:10:15.702920 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.702455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.129:45547)
0620 20:10:15.702690 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.702775 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 20:10:15.707566 12312 mvcc.cc:205] Tried to move back new op lower bound from 7299015539496927232 to 7299015539213074432. Current Snapshot: MvccSnapshot[applied={T|T < 7299015539496927232}]
I20260620 20:10:15.708986 12313 mvcc.cc:205] Tried to move back new op lower bound from 7299015539496927232 to 7299015539213074432. Current Snapshot: MvccSnapshot[applied={T|T < 7299015539496927232}]
I20260620 20:10:15.709873 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.709507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.129:50427)
0620 20:10:15.709653 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.709735 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 20:10:15.713414 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.712989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.130:56433)
0620 20:10:15.713208 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.713283 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 20:10:15.714831 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.714286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:54629)
0620 20:10:15.714526 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.714626 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260620 20:10:15.720557 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.718854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.4.169.131:55029)
0620 20:10:15.720284 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.720390 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1280}
I20260620 20:10:15.726604 11533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:15.726861 11533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:15.730146 12314 mvcc.cc:205] Tried to move back new op lower bound from 7299015539496927232 to 7299015539213074432. Current Snapshot: MvccSnapshot[applied={T|T < 7299015539496927232 or (T in {7299015539496927232})}]
I20260620 20:10:15.734320 11533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc5844a44fc04b1eb9479d71b25ace63: Sending DeleteTablet for 3 replicas of tablet 6634631c6cfb4efeb8947f3263a8cf1d
I20260620 20:10:15.735301 11837 tablet_service.cc:1514] Processing DeleteTablet for tablet 6634631c6cfb4efeb8947f3263a8cf1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:15 UTC) from {username='oryx', principal='oryx/127.4.169.189@KRBTEST.COM'} at 127.0.0.1:57054
I20260620 20:10:15.735491 11983 tablet_service.cc:1514] Processing DeleteTablet for tablet 6634631c6cfb4efeb8947f3263a8cf1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:15 UTC) from {username='oryx', principal='oryx/127.4.169.189@KRBTEST.COM'} at 127.0.0.1:53422
I20260620 20:10:15.735602 11982 tablet_service.cc:1514] Processing DeleteTablet for tablet 6634631c6cfb4efeb8947f3263a8cf1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:15 UTC) from {username='oryx', principal='oryx/127.4.169.189@KRBTEST.COM'} at 127.0.0.1:53422
I20260620 20:10:15.736261 12322 tablet_replica.cc:330] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: stopping tablet replica
I20260620 20:10:15.736488 12322 raft_consensus.cc:2229] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:15.736774 12124 tablet_service.cc:1514] Processing DeleteTablet for tablet 6634631c6cfb4efeb8947f3263a8cf1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:15 UTC) from {username='oryx', principal='oryx/127.4.169.189@KRBTEST.COM'} at 127.0.0.1:43976
I20260620 20:10:15.736851 12322 raft_consensus.cc:2258] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:15.738052 12322 ts_tablet_manager.cc:1905] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:15.738317 12323 tablet_replica.cc:330] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: stopping tablet replica
I20260620 20:10:15.738528 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 11700
I20260620 20:10:15.738528 12323 raft_consensus.cc:2229] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:10:15.739199 12323 raft_consensus.cc:2258] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:15.739805 12321 tablet_replica.cc:330] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: stopping tablet replica
I20260620 20:10:15.739988 12321 raft_consensus.cc:2229] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:15.740312 12321 raft_consensus.cc:2258] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:15.742183 12321 ts_tablet_manager.cc:1905] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:15.745673 12323 ts_tablet_manager.cc:1905] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:15.746447 12322 ts_tablet_manager.cc:1918] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:15.746639 12322 log.cc:1196] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/6634631c6cfb4efeb8947f3263a8cf1d
I20260620 20:10:15.747246 12322 ts_tablet_manager.cc:1939] T 6634631c6cfb4efeb8947f3263a8cf1d P f359aa5fee0d4e49949dc64a62e96dbb: Deleting consensus metadata
I20260620 20:10:15.748574 11519 catalog_manager.cc:4841] TS f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699): tablet 6634631c6cfb4efeb8947f3263a8cf1d (table test-table [id=6e985c01771d460b899b83dceac088b6]) successfully deleted
I20260620 20:10:15.748929 12321 ts_tablet_manager.cc:1918] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:15.749166 12321 log.cc:1196] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/6634631c6cfb4efeb8947f3263a8cf1d
W20260620 20:10:15.749624 11519 catalog_manager.cc:4808] TS f359aa5fee0d4e49949dc64a62e96dbb (127.4.169.130:41699): delete failed for tablet 6634631c6cfb4efeb8947f3263a8cf1d because the tablet was not found. No further retry: Not found: Tablet not found: 6634631c6cfb4efeb8947f3263a8cf1d
I20260620 20:10:15.749907 12321 ts_tablet_manager.cc:1939] T 6634631c6cfb4efeb8947f3263a8cf1d P c3e85269446e48feb97c9899cc44b756: Deleting consensus metadata
I20260620 20:10:15.751197 11518 catalog_manager.cc:4841] TS c3e85269446e48feb97c9899cc44b756 (127.4.169.129:35319): tablet 6634631c6cfb4efeb8947f3263a8cf1d (table test-table [id=6e985c01771d460b899b83dceac088b6]) successfully deleted
I20260620 20:10:15.754235 12323 ts_tablet_manager.cc:1918] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:15.754412 12323 log.cc:1196] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/6634631c6cfb4efeb8947f3263a8cf1d
I20260620 20:10:15.754922 12323 ts_tablet_manager.cc:1939] T 6634631c6cfb4efeb8947f3263a8cf1d P 23d7e639f3ee49efab35348707f2a64e: Deleting consensus metadata
I20260620 20:10:15.756134 11517 catalog_manager.cc:4841] TS 23d7e639f3ee49efab35348707f2a64e (127.4.169.131:40573): tablet 6634631c6cfb4efeb8947f3263a8cf1d (table test-table [id=6e985c01771d460b899b83dceac088b6]) successfully deleted
I20260620 20:10:15.770573 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.739544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.4.169.129:35319)
0620 20:10:15.740142 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:15.740150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:15.756351 (+ 16201us) server_negotiation.cc:407] Connection header received
0620 20:10:15.756535 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:15.756538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:15.756605 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:15.756682 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:15.758486 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.759195 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.760948 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.761341 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.761428 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.764764 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:15.764779 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:15.764793 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:15.764823 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:15.767604 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:15.768370 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:15.768376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:15.768378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:15.768499 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:15.768879 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:15.768882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:15.768884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:15.769314 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:15.769561 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:15.769916 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 20:10:15.770157 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454}
I20260620 20:10:15.771245 11897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:15.782989 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.782477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55638)
0620 20:10:15.782670 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.782786 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 20:10:15.784158 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.783716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55648)
0620 20:10:15.783923 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:15.784017 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 20:10:15.891330 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 11700
I20260620 20:10:15.922757 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 11914
I20260620 20:10:15.937404 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:15.923555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.4.169.130:41699)
0620 20:10:15.923719 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:15.923724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:15.923820 (+ 96us) server_negotiation.cc:407] Connection header received
0620 20:10:15.923988 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:15.923992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:15.924054 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:15.924137 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:15.925816 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.926669 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.928439 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:15.928819 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:15.928909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:15.931941 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:15.931955 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:15.931969 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:15.932004 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:15.934795 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:15.935582 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:15.935589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:15.935591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:15.935709 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:15.936076 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:15.936079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:15.936081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:15.936381 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:15.936628 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:15.936929 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 20:10:15.937089 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:10:15.938092 12043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:16.075647 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 11914
I20260620 20:10:16.098527 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 12061
I20260620 20:10:16.113155 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:16.099372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.4.169.131:40573)
0620 20:10:16.099546 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:16.099553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:16.099597 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:10:16.099835 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:16.099839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:16.099929 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:16.100038 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:16.101321 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:16.102134 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:16.103435 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:16.103874 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:16.104033 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:16.106749 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:16.106763 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:16.106779 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:16.106813 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:16.110067 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:16.110964 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:16.110973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:16.110977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:16.111122 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:16.111530 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:16.111541 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:16.111543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:16.111913 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:16.112259 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:16.112570 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 20:10:16.112808 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:10:16.113847 12184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:16.241391 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 12061
I20260620 20:10:16.265904 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 11395
I20260620 20:10:16.267009 11563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:16.387671 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:16.386883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38587 (local address 127.0.0.1:55654)
0620 20:10:16.387370 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:16.387481 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:38587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":282,"threads_started":1}
I20260620 20:10:16.387634 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 11395
I20260620 20:10:16.408499 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 11496
I20260620 20:10:16.409551 11753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:16.526566 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 11496
2026-06-20T20:10:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11480 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:10:16 dist-test-slave-4bwq krb5kdc[12333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:10:16 dist-test-slave-4bwq krb5kdc[12333](info): set up 2 sockets
Jun 20 20:10:16 dist-test-slave-4bwq krb5kdc[12333](info): commencing operation
krb5kdc: starting...
W20260620 20:10:17.615319 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:10:17 dist-test-slave-4bwq krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:10:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.190.keytab.
Entry for principal oryx/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.190.keytab.
I20260620 20:10:17.823781 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42139
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.190.keytab
--principal=oryx/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:42139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:18.141590 12346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:18.141886 12346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:18.142066 12346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:18.149410 12346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:18.149523 12346 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:10:18.149609 12346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:18.149724 12346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:18.149804 12346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:18.149897 12346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:18.159840 12346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:42139
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.190.keytab
--principal=oryx/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:18.161151 12346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:18.162549 12346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:18.174324 12352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:18.174423 12354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:18.175081 12351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:18.175082 12346 server_base.cc:1034] running on GCE node
I20260620 20:10:18.175961 12346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:18.177289 12346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:18.178508 12346 hybrid_clock.cc:648] HybridClock initialized: now 1781986218178442 us; error 31 us; skew 500 ppm
I20260620 20:10:18.178851 12346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:18 dist-test-slave-4bwq krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986218, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:18.182088 12346 init.cc:377] Logged in from keytab as oryx/127.4.169.190@KRBTEST.COM (short username oryx)
I20260620 20:10:18.183663 12346 webserver.cc:460] Webserver started at http://127.4.169.190:36901/ using document root <none> and password file <none>
I20260620 20:10:18.184403 12346 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:18.184528 12346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:18.184857 12346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:18.187944 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "51c4eb61385444ca8c1d7016e661539b"
format_stamp: "Formatted at 2026-06-20 20:10:18 on dist-test-slave-4bwq"
server_key: "3d9b59c63fdd41987e786f7aded278a5"
server_key_iv: "485051953c87cb0738eaf6cc7f085f9f"
server_key_version: "encryptionkey@0"
I20260620 20:10:18.188778 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "51c4eb61385444ca8c1d7016e661539b"
format_stamp: "Formatted at 2026-06-20 20:10:18 on dist-test-slave-4bwq"
server_key: "3d9b59c63fdd41987e786f7aded278a5"
server_key_iv: "485051953c87cb0738eaf6cc7f085f9f"
server_key_version: "encryptionkey@0"
I20260620 20:10:18.193918 12346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:10:18.197391 12361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:18.198771 12346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:10:18.198937 12346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "51c4eb61385444ca8c1d7016e661539b"
format_stamp: "Formatted at 2026-06-20 20:10:18 on dist-test-slave-4bwq"
server_key: "3d9b59c63fdd41987e786f7aded278a5"
server_key_iv: "485051953c87cb0738eaf6cc7f085f9f"
server_key_version: "encryptionkey@0"
I20260620 20:10:18.199150 12346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:18.224964 12346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:18.229242 12346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:18.229513 12346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:18.250540 12346 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:42139
I20260620 20:10:18.250548 12422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:42139 every 8 connection(s)
I20260620 20:10:18.251844 12346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:10:18.255518 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:18.262230 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 12346
I20260620 20:10:18.262506 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:10:18.263283 4774 external_mini_cluster.cc:1428] Setting key 17b173ec15f76bb254524550f4f8528f
I20260620 20:10:18.266476 12423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b: Bootstrap starting.
I20260620 20:10:18.269891 12423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:18.271054 12423 log.cc:824] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:18.274114 12423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b: No bootstrap required, opened a new log
Jun 20 20:10:18 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:18.284912 12423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c4eb61385444ca8c1d7016e661539b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42139 } }
I20260620 20:10:18.285387 12423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:18.285503 12423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c4eb61385444ca8c1d7016e661539b, State: Initialized, Role: FOLLOWER
I20260620 20:10:18.286209 12423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c4eb61385444ca8c1d7016e661539b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42139 } }
I20260620 20:10:18.286607 12423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:10:18.286767 12423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:10:18.286949 12423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:18.288338 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:18.265309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.4.169.190:42139)
0620 20:10:18.266098 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:18.266110 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:18.266151 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:10:18.267928 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:18.267952 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:18.268334 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:18.268639 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:18.270446 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.272037 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.273750 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.274490 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.274623 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:18.279127 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:18.279142 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:18.279162 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:18.279232 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:18.283475 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:18.284190 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:18.284198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:18.284207 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:18.284409 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:18.284765 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:18.284770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:18.284773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:18.285531 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:18.285816 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:18.286344 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 20:10:18.286848 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":386,"threads_started":1}
I20260620 20:10:18.289796 12423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c4eb61385444ca8c1d7016e661539b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42139 } }
I20260620 20:10:18.290303 12423 leader_election.cc:302] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51c4eb61385444ca8c1d7016e661539b; no voters:
I20260620 20:10:18.291038 12423 leader_election.cc:288] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:10:18.291157 12428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:18.291885 12428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [term 1 LEADER]: Becoming Leader. State: Replica: 51c4eb61385444ca8c1d7016e661539b, State: Running, Role: LEADER
I20260620 20:10:18.292521 12428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c4eb61385444ca8c1d7016e661539b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42139 } }
I20260620 20:10:18.293294 12423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:18.295213 12430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51c4eb61385444ca8c1d7016e661539b. Latest consensus state: current_term: 1 leader_uuid: "51c4eb61385444ca8c1d7016e661539b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c4eb61385444ca8c1d7016e661539b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42139 } } }
I20260620 20:10:18.295590 12430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:18.295822 12429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51c4eb61385444ca8c1d7016e661539b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c4eb61385444ca8c1d7016e661539b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42139 } } }
I20260620 20:10:18.296206 12429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:18.297737 12435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:18.302728 12435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:18.311951 12435 catalog_manager.cc:1347] Generated new cluster ID: 3c106f5311f84cd79d565ccb70f1760f
I20260620 20:10:18.312057 12435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:18.328994 12435 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:10:18.330312 12435 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:18.339460 12435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b: Generated new TSK 0
I20260620 20:10:18.340238 12435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab.
Entry for principal oryx/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab.
I20260620 20:10:18.404049 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42139
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab
--principal=oryx/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:18.729928 12449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:18.730257 12449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:18.730485 12449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:18.737701 12449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:18.737856 12449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:18.738013 12449 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:10:18.738157 12449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:10:18.747974 12449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab
--principal=oryx/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.169.190:42139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:18.749347 12449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:18.750713 12449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:18.763553 12457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:18.763618 12454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:18.763592 12455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:18.764384 12449 server_base.cc:1034] running on GCE node
I20260620 20:10:18.764995 12449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:18.765919 12449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:18.767155 12449 hybrid_clock.cc:648] HybridClock initialized: now 1781986218767087 us; error 48 us; skew 500 ppm
I20260620 20:10:18.767457 12449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:18 dist-test-slave-4bwq krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986218, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:18.770678 12449 init.cc:377] Logged in from keytab as oryx/127.4.169.129@KRBTEST.COM (short username oryx)
I20260620 20:10:18.772068 12449 webserver.cc:460] Webserver started at http://127.4.169.129:45973/ using document root <none> and password file <none>
I20260620 20:10:18.772816 12449 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:18.772936 12449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:18.773247 12449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:18.776173 12449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "dda99e976e36442baf918c1d43e6edc0"
format_stamp: "Formatted at 2026-06-20 20:10:18 on dist-test-slave-4bwq"
server_key: "5ca1effb147648e6bf4a428555bc910e"
server_key_iv: "864591b151f464492d380402d1c9a6cc"
server_key_version: "encryptionkey@0"
I20260620 20:10:18.777063 12449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "dda99e976e36442baf918c1d43e6edc0"
format_stamp: "Formatted at 2026-06-20 20:10:18 on dist-test-slave-4bwq"
server_key: "5ca1effb147648e6bf4a428555bc910e"
server_key_iv: "864591b151f464492d380402d1c9a6cc"
server_key_version: "encryptionkey@0"
I20260620 20:10:18.782024 12449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:10:18.785473 12464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:18.786875 12449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:18.787052 12449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "dda99e976e36442baf918c1d43e6edc0"
format_stamp: "Formatted at 2026-06-20 20:10:18 on dist-test-slave-4bwq"
server_key: "5ca1effb147648e6bf4a428555bc910e"
server_key_iv: "864591b151f464492d380402d1c9a6cc"
server_key_version: "encryptionkey@0"
I20260620 20:10:18.787300 12449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:18.802673 12449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:18.807168 12449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:18.807439 12449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:18.810119 12449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:18.810230 12449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:18.810539 12449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:18.810652 12449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 20:10:18 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986218, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:18.833644 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:18.811125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:48207 (local address 127.4.169.190:42139)
0620 20:10:18.811721 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:18.811727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:18.812939 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 20:10:18.815616 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:18.815623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:18.815740 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:18.815873 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:18.818056 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.819403 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.821643 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.822429 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.822588 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:18.826536 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:18.826546 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:18.826554 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:18.826601 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:18.829939 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:18.830908 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:18.830916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:18.830920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:18.831042 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:18.831563 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:18.831568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:18.831571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:18.831932 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:18.832082 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:18.833049 (+ 967us) server_negotiation.cc:299] Negotiation successful
0620 20:10:18.833262 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":364,"threads_started":1}
I20260620 20:10:18.834860 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:18.811850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42139 (local address 127.4.169.129:48207)
0620 20:10:18.812755 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:18.812798 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:18.815245 (+ 2447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:18.816165 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:18.816181 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:18.817026 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:18.817823 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:18.817868 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.819585 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.819596 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:18.821466 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:18.821478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.822690 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.822704 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:18.822920 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:18.823719 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:18.823741 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:18.826235 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:18.830074 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:18.830083 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:18.830105 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:18.830784 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:18.831147 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:18.831153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:18.831158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:18.831452 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:18.832159 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:18.832165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:18.832648 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 20:10:18.832969 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 20:10:18.833501 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":404,"threads_started":1}
I20260620 20:10:18.856106 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:18.840776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42139 (local address 127.4.169.129:41321)
0620 20:10:18.841493 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:18.841517 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:18.841690 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:18.842020 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:18.842027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:18.842826 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:18.843444 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:18.843467 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.844451 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.844460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:18.846632 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:18.846644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.847429 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.847438 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:18.847727 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:18.848510 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:18.848540 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:18.849591 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:18.853105 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:18.853117 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:18.853121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:18.853521 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:18.853929 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:18.853934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:18.853938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:18.854055 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:18.854674 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:18.854679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:18.854906 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 20:10:18.855113 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 20:10:18.855508 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":383,"threads_started":1}
I20260620 20:10:18.856671 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:18.840775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:41321 (local address 127.4.169.190:42139)
0620 20:10:18.840976 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:18.840983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:18.841609 (+ 626us) server_negotiation.cc:407] Connection header received
0620 20:10:18.841817 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:18.841822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:18.841885 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:18.841964 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:18.843613 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.844315 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.846777 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:18.847204 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:18.847340 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:18.849856 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:18.849864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:18.849871 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:18.849911 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:18.852936 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:18.853667 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:18.853672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:18.853692 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:18.853819 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:18.854164 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:18.854171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:18.854176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:18.854573 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:18.854717 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:18.856162 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0620 20:10:18.856411 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:10:18.867861 12449 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:39639
I20260620 20:10:18.867884 12584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:39639 every 8 connection(s)
I20260620 20:10:18.869108 12449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:10:18.871199 12585 heartbeater.cc:347] Connected to a master server at 127.4.169.190:42139
I20260620 20:10:18.871426 12585 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:18.872083 12585 heartbeater.cc:511] Master 127.4.169.190:42139 requested a full tablet report, sending...
I20260620 20:10:18.873991 12378 ts_manager.cc:194] Registered new tserver with Master: dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639)
I20260620 20:10:18.875509 12378 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:48207
I20260620 20:10:18.877743 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 12449
I20260620 20:10:18.877957 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:10:18.878654 4774 external_mini_cluster.cc:1428] Setting key 768bc5d13e5c62cc956068af7f96bb24
WARNING: no policy specified for oryx/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab.
Entry for principal oryx/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab.
I20260620 20:10:18.922597 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42139
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab
--principal=oryx/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:19.245096 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:19.245410 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:19.245579 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:19.252796 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:19.252939 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:19.253087 12589 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:10:19.253227 12589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:10:19.262688 12589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab
--principal=oryx/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.169.190:42139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:19.263904 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:19.265237 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:19.278074 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:19.278007 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:19.278046 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:19.279165 12589 server_base.cc:1034] running on GCE node
I20260620 20:10:19.279757 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:19.280680 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:19.281913 12589 hybrid_clock.cc:648] HybridClock initialized: now 1781986219281825 us; error 68 us; skew 500 ppm
I20260620 20:10:19.282202 12589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:19 dist-test-slave-4bwq krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986219, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:19.285329 12589 init.cc:377] Logged in from keytab as oryx/127.4.169.130@KRBTEST.COM (short username oryx)
I20260620 20:10:19.286832 12589 webserver.cc:460] Webserver started at http://127.4.169.130:38237/ using document root <none> and password file <none>
I20260620 20:10:19.287541 12589 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:19.287672 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:19.287978 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:19.291023 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "dc74bc6d06de4547bac7512514c33ad5"
format_stamp: "Formatted at 2026-06-20 20:10:19 on dist-test-slave-4bwq"
server_key: "0fcfed3893d77caaafb5c0eaaee0d2e9"
server_key_iv: "9c0fc86faddf4ee0e81357f9c5777ca7"
server_key_version: "encryptionkey@0"
I20260620 20:10:19.291965 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "dc74bc6d06de4547bac7512514c33ad5"
format_stamp: "Formatted at 2026-06-20 20:10:19 on dist-test-slave-4bwq"
server_key: "0fcfed3893d77caaafb5c0eaaee0d2e9"
server_key_iv: "9c0fc86faddf4ee0e81357f9c5777ca7"
server_key_version: "encryptionkey@0"
I20260620 20:10:19.296980 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:10:19.300556 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:19.301946 12589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:19.302112 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "dc74bc6d06de4547bac7512514c33ad5"
format_stamp: "Formatted at 2026-06-20 20:10:19 on dist-test-slave-4bwq"
server_key: "0fcfed3893d77caaafb5c0eaaee0d2e9"
server_key_iv: "9c0fc86faddf4ee0e81357f9c5777ca7"
server_key_version: "encryptionkey@0"
I20260620 20:10:19.302330 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:19.320350 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:19.324074 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:19.324327 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:19.326835 12589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:19.326934 12589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:19.327111 12589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:19.327205 12589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 20:10:19 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986219, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:19.355355 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:19.327305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:36595 (local address 127.4.169.190:42139)
0620 20:10:19.327524 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:19.327535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:19.328948 (+ 1413us) server_negotiation.cc:407] Connection header received
0620 20:10:19.332005 (+ 3057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:19.332011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:19.332106 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:19.332201 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:19.335075 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.336148 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.338350 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.339168 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.339327 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:19.348797 (+ 9470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:19.348802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:19.348806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:19.348867 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:19.352264 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:19.352991 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:19.352997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:19.353000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:19.353122 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:19.353551 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:19.353556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:19.353559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:19.353851 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:19.354002 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:19.354787 (+ 785us) server_negotiation.cc:299] Negotiation successful
0620 20:10:19.355007 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:10:19.356473 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:19.327920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42139 (local address 127.4.169.130:36595)
0620 20:10:19.328759 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:19.328809 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:19.331576 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:19.332820 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:19.332835 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:19.333805 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:19.334855 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:19.334887 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.336333 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.336338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:19.338179 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:19.338198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.339426 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.339435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:19.339803 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:19.344952 (+ 5149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:19.344974 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:19.348474 (+ 3500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:19.352401 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:19.352409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:19.352429 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:19.352863 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:19.353233 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:19.353238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:19.353242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:19.353435 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:19.353976 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:19.353984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:19.354342 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 20:10:19.354675 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 20:10:19.355227 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":346,"threads_started":1}
I20260620 20:10:19.376582 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:19.362076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:55157 (local address 127.4.169.190:42139)
0620 20:10:19.362314 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:19.362323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:19.362867 (+ 544us) server_negotiation.cc:407] Connection header received
0620 20:10:19.363006 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:19.363015 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:19.363111 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:19.363240 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:19.364050 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.365377 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.367375 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.367796 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.367932 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:19.370897 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:19.370901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:19.370904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:19.370932 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:19.373489 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:19.374554 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:19.374562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:19.374567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:19.374734 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:19.375123 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:19.375127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:19.375131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:19.375511 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:19.375672 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:19.376204 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 20:10:19.376366 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:10:19.377188 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:19.362089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42139 (local address 127.4.169.130:55157)
0620 20:10:19.362778 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:19.362794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:19.362901 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:19.363263 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:19.363266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:19.363653 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:19.363923 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:19.363930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.365540 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.365551 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:19.367231 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:19.367244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.367989 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.367995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:19.368170 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:19.368855 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:19.368890 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:19.370466 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:19.373662 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:19.373670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:19.373677 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:19.374415 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:19.374880 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:19.374883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:19.374886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:19.375003 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:19.375657 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:19.375665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:19.375929 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 20:10:19.376107 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 20:10:19.376530 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":403,"threads_started":1}
I20260620 20:10:19.385629 12589 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:43333
I20260620 20:10:19.385694 12723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:43333 every 8 connection(s)
I20260620 20:10:19.386972 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:10:19.389115 12724 heartbeater.cc:347] Connected to a master server at 127.4.169.190:42139
I20260620 20:10:19.389319 12724 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:19.389932 12724 heartbeater.cc:511] Master 127.4.169.190:42139 requested a full tablet report, sending...
I20260620 20:10:19.391268 12378 ts_manager.cc:194] Registered new tserver with Master: dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333)
I20260620 20:10:19.392117 12378 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:36595
I20260620 20:10:19.393167 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 12589
I20260620 20:10:19.393374 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:10:19.393785 4774 external_mini_cluster.cc:1428] Setting key 25e5c712b9fd5680859feac084caf8c3
WARNING: no policy specified for oryx/127.4.169.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab.
Entry for principal oryx/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab.
I20260620 20:10:19.441068 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42139
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab
--principal=oryx/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:19.769578 12728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:19.769842 12728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:19.769937 12728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:19.777143 12728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:19.777252 12728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:19.777334 12728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:10:19.777415 12728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:10:19.787002 12728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab
--principal=oryx/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.169.190:42139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:19.788206 12728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:19.789482 12728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:19.802493 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:19.802567 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:19.802680 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:19.803261 12728 server_base.cc:1034] running on GCE node
I20260620 20:10:19.803987 12728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:19.804940 12728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:19.806229 12728 hybrid_clock.cc:648] HybridClock initialized: now 1781986219806120 us; error 63 us; skew 500 ppm
I20260620 20:10:19.806620 12728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:19 dist-test-slave-4bwq krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986219, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:19.809965 12728 init.cc:377] Logged in from keytab as oryx/127.4.169.131@KRBTEST.COM (short username oryx)
I20260620 20:10:19.811462 12728 webserver.cc:460] Webserver started at http://127.4.169.131:36413/ using document root <none> and password file <none>
I20260620 20:10:19.812269 12728 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:19.812363 12728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:19.812633 12728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:19.815639 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "7745080856f8400d893c44688e5d00b7"
format_stamp: "Formatted at 2026-06-20 20:10:19 on dist-test-slave-4bwq"
server_key: "7e621632ace96a9b1d3304df4b47e6ef"
server_key_iv: "fbe2ab1ebdb0750d0245d827c5380b8d"
server_key_version: "encryptionkey@0"
I20260620 20:10:19.816403 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "7745080856f8400d893c44688e5d00b7"
format_stamp: "Formatted at 2026-06-20 20:10:19 on dist-test-slave-4bwq"
server_key: "7e621632ace96a9b1d3304df4b47e6ef"
server_key_iv: "fbe2ab1ebdb0750d0245d827c5380b8d"
server_key_version: "encryptionkey@0"
I20260620 20:10:19.822086 12728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:10:19.825747 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:19.827342 12728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:19.827510 12728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "7745080856f8400d893c44688e5d00b7"
format_stamp: "Formatted at 2026-06-20 20:10:19 on dist-test-slave-4bwq"
server_key: "7e621632ace96a9b1d3304df4b47e6ef"
server_key_iv: "fbe2ab1ebdb0750d0245d827c5380b8d"
server_key_version: "encryptionkey@0"
I20260620 20:10:19.827749 12728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:19.850553 12728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:19.855131 12728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:19.855412 12728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:19.858314 12728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:19.858438 12728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:10:19.858611 12728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:19.858738 12728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:10:19 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986219, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:19.878510 12585 heartbeater.cc:503] Master 127.4.169.190:42139 was elected leader, sending a full tablet report...
I20260620 20:10:19.884888 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:19.858876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:33759 (local address 127.4.169.190:42139)
0620 20:10:19.859056 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:19.859064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:19.860574 (+ 1510us) server_negotiation.cc:407] Connection header received
0620 20:10:19.862653 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:19.862659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:19.862756 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:19.862904 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:19.865219 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.866264 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.870112 (+ 3848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.870802 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.870944 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:19.876470 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:19.876480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:19.876488 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:19.876548 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:19.880989 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:19.882078 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:19.882086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:19.882092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:19.882260 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:19.882832 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:19.882840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:19.882844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:19.883116 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:19.883260 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:19.884289 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0620 20:10:19.884511 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:10:19.886518 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:19.859576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42139 (local address 127.4.169.131:33759)
0620 20:10:19.860404 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:19.860448 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:19.862266 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:19.863174 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:19.863183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:19.864015 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:19.865037 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:19.865065 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.866642 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.866655 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:19.869854 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:19.869884 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.871118 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.871129 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:19.871513 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:19.872472 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:19.872516 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:19.876142 (+ 3626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:19.881149 (+ 5007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:19.881159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:19.881178 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:19.881937 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:19.882472 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:19.882479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:19.882485 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:19.882703 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:19.883255 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:19.883263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:19.883814 (+ 551us) client_negotiation.cc:769] Sending connection context
0620 20:10:19.884150 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 20:10:19.884751 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":378,"threads_started":1}
I20260620 20:10:19.908151 12728 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:42425
I20260620 20:10:19.908170 12862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:42425 every 8 connection(s)
I20260620 20:10:19.909888 12728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:10:19.912045 12863 heartbeater.cc:347] Connected to a master server at 127.4.169.190:42139
I20260620 20:10:19.912273 12863 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:19.912990 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 12728
I20260620 20:10:19.913048 12863 heartbeater.cc:511] Master 127.4.169.190:42139 requested a full tablet report, sending...
I20260620 20:10:19.913195 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:10:19.913724 4774 external_mini_cluster.cc:1428] Setting key 54483c1886c340b137192ef5616dccc5
I20260620 20:10:19.914916 12378 ts_manager.cc:194] Registered new tserver with Master: 7745080856f8400d893c44688e5d00b7 (127.4.169.131:42425)
I20260620 20:10:19.915793 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:19.901139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42139 (local address 127.4.169.131:53335)
0620 20:10:19.901978 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:19.902007 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:19.902210 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:19.902678 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:19.902683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:19.903360 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:19.903918 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:19.903938 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.905272 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.905281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:19.906707 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:19.906719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.907683 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.907693 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:19.907869 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:19.908551 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:19.908574 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:19.909634 (+ 1060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:19.913132 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:19.913139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:19.913143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:19.913537 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:19.913913 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:19.913919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:19.913922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:19.914042 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:19.914699 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:19.914704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:19.914892 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 20:10:19.915049 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 20:10:19.915359 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":404,"threads_started":1}
I20260620 20:10:19.916008 12378 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:33759
I20260620 20:10:19.916676 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:19.901144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:53335 (local address 127.4.169.190:42139)
0620 20:10:19.901366 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:19.901374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:19.902132 (+ 758us) server_negotiation.cc:407] Connection header received
0620 20:10:19.902380 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:19.902417 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:19.902500 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:19.902614 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:19.904068 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.905107 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.906845 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.907466 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.907596 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:19.909813 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:19.909818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:19.909821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:19.909851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:19.913008 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:19.913661 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:19.913666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:19.913670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:19.913810 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:19.914152 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:19.914157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:19.914161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:19.914566 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:19.914704 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:19.916075 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0620 20:10:19.916294 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:10:19.916884 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:10:19 dist-test-slave-4bwq krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:10:19 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:19 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:20.394562 12724 heartbeater.cc:503] Master 127.4.169.190:42139 was elected leader, sending a full tablet report...
I20260620 20:10:20.918746 12863 heartbeater.cc:503] Master 127.4.169.190:42139 was elected leader, sending a full tablet report...
W20260620 20:10:21.041968 12871 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.079s user 0.005s sys 0.010s
W20260620 20:10:21.043286 12473 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:19.956390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.4.169.190:42139)
0620 20:10:19.956618 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:19.956625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:19.956837 (+ 212us) server_negotiation.cc:407] Connection header received
0620 20:10:19.957059 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:19.957069 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:19.957133 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:19.957249 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:19.958722 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.959562 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.961018 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:19.961456 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:19.961586 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.042963 (+1081377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33598: BlockingRecv error: recv got EOF from 127.0.0.1:33598 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 20:10:21 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:21.060346 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.047135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.4.169.190:42139)
0620 20:10:21.047317 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.047321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.047674 (+ 353us) server_negotiation.cc:407] Connection header received
0620 20:10:21.047846 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.047849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.047953 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.048037 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:21.049268 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.049984 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.051284 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.051678 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.051775 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.055383 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:21.055387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:21.055390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:21.055420 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:21.057754 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:21.058437 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:21.058447 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:21.058449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:21.058560 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:21.058946 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:21.058952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:21.058956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:21.059235 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:21.059563 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.059824 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.060036 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:10:21.064296 12377 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:10:21.071208 12377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:10:21.107077 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.084022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.4.169.130:43333)
0620 20:10:21.095272 (+ 11250us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.095286 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.095972 (+ 686us) server_negotiation.cc:407] Connection header received
0620 20:10:21.096108 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.096121 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.096490 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.099202 (+ 2712us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:21.099469 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.101104 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.104447 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.105972 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.106123 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.106420 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.106616 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.106787 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11004,"thread_start_us":10836,"threads_started":1}
I20260620 20:10:21.108369 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.083523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:43333 (local address 127.0.0.1:46598)
0620 20:10:21.095611 (+ 12088us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.095638 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.095806 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.096810 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.096822 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.096865 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:21.098928 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.098947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.101268 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.101279 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.104285 (+ 3006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.104301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.106159 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.106165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.106347 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.106419 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.106572 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.106758 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11894,"mutex_wait_us":23,"thread_start_us":16901,"threads_started":1}
I20260620 20:10:21.108656 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.083040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:42425 (local address 127.0.0.1:41434)
0620 20:10:21.084519 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.084586 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.084799 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.100996 (+ 16197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.101001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.101024 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:21.101475 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.101496 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.103412 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.103417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.105639 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.105653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.107763 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.107769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.107927 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.107960 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.108062 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.108193 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":68,"thread_start_us":916,"threads_started":1}
I20260620 20:10:21.108705 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.083283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.4.169.131:42425)
0620 20:10:21.100380 (+ 17097us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.100392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.100424 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:10:21.100537 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.100546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.100812 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.101113 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:21.101647 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.103271 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.105796 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.107535 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.107672 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.107865 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.108123 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.108278 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16939,"thread_start_us":16810,"threads_started":1}
I20260620 20:10:21.109896 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.085270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.4.169.129:39639)
0620 20:10:21.100394 (+ 15124us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.100403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.100802 (+ 399us) server_negotiation.cc:407] Connection header received
0620 20:10:21.100944 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.100952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.101188 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.101387 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:21.102081 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.103442 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.105553 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.107131 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.107272 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.108460 (+ 1188us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.108527 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.108686 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14569,"thread_start_us":14428,"threads_started":1}
I20260620 20:10:21.110157 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.084963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39639 (local address 127.0.0.1:60000)
0620 20:10:21.100671 (+ 15708us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.100699 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.100818 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.101400 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.101406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.101446 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:21.101922 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.101933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.103588 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.103594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.105407 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.105419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.107535 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.107542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.107723 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.107757 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.107831 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.107965 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15605,"mutex_wait_us":1,"thread_start_us":10315,"threads_started":1}
I20260620 20:10:21.112521 12793 tablet_service.cc:1467] Processing CreateTablet for tablet ee24569851df4b23815ecf807439805d (DEFAULT_TABLE table=test-table [id=a6ea7e2a850c4f25a2bfc38edb8f7230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:21.112521 12654 tablet_service.cc:1467] Processing CreateTablet for tablet ee24569851df4b23815ecf807439805d (DEFAULT_TABLE table=test-table [id=a6ea7e2a850c4f25a2bfc38edb8f7230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:21.112829 12515 tablet_service.cc:1467] Processing CreateTablet for tablet ee24569851df4b23815ecf807439805d (DEFAULT_TABLE table=test-table [id=a6ea7e2a850c4f25a2bfc38edb8f7230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:21.113953 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee24569851df4b23815ecf807439805d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:21.113951 12654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee24569851df4b23815ecf807439805d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:21.114183 12515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee24569851df4b23815ecf807439805d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:21.124958 12888 tablet_bootstrap.cc:492] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: Bootstrap starting.
I20260620 20:10:21.128280 12888 tablet_bootstrap.cc:654] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:21.128301 12889 tablet_bootstrap.cc:492] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: Bootstrap starting.
I20260620 20:10:21.130820 12888 log.cc:824] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:21.131106 12890 tablet_bootstrap.cc:492] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: Bootstrap starting.
I20260620 20:10:21.131927 12889 tablet_bootstrap.cc:654] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:21.133432 12889 log.cc:824] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:21.134130 12890 tablet_bootstrap.cc:654] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:21.135458 12890 log.cc:824] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:21.136443 12889 tablet_bootstrap.cc:492] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: No bootstrap required, opened a new log
I20260620 20:10:21.136701 12888 tablet_bootstrap.cc:492] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: No bootstrap required, opened a new log
I20260620 20:10:21.136731 12889 ts_tablet_manager.cc:1397] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:10:21.136976 12888 ts_tablet_manager.cc:1397] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 20:10:21.138373 12890 tablet_bootstrap.cc:492] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: No bootstrap required, opened a new log
I20260620 20:10:21.138645 12890 ts_tablet_manager.cc:1397] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 20:10:21.144515 12889 raft_consensus.cc:348] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } }
I20260620 20:10:21.145020 12889 raft_consensus.cc:374] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:21.145129 12889 raft_consensus.cc:729] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7745080856f8400d893c44688e5d00b7, State: Initialized, Role: FOLLOWER
I20260620 20:10:21.145735 12889 consensus_queue.cc:260] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } }
I20260620 20:10:21.147554 12889 ts_tablet_manager.cc:1428] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260620 20:10:21.147662 12888 raft_consensus.cc:348] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } }
I20260620 20:10:21.148094 12888 raft_consensus.cc:374] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:21.148190 12888 raft_consensus.cc:729] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda99e976e36442baf918c1d43e6edc0, State: Initialized, Role: FOLLOWER
I20260620 20:10:21.148756 12888 consensus_queue.cc:260] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } }
I20260620 20:10:21.148913 12890 raft_consensus.cc:348] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } }
I20260620 20:10:21.149413 12890 raft_consensus.cc:374] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:21.149554 12890 raft_consensus.cc:729] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc74bc6d06de4547bac7512514c33ad5, State: Initialized, Role: FOLLOWER
I20260620 20:10:21.150197 12890 consensus_queue.cc:260] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } }
I20260620 20:10:21.150441 12888 ts_tablet_manager.cc:1428] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 20:10:21.151822 12890 ts_tablet_manager.cc:1428] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260620 20:10:21.162319 12864 tablet.cc:2369] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:10:21.372527 12586 tablet.cc:2369] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:10:21.390062 12725 tablet.cc:2369] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:10:21.485827 12896 raft_consensus.cc:482] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:10:21.486171 12896 raft_consensus.cc:504] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } }
I20260620 20:10:21.487869 12896 leader_election.cc:288] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7745080856f8400d893c44688e5d00b7 (127.4.169.131:42425), dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639)
I20260620 20:10:21.495115 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.488341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:46463 (local address 127.4.169.131:42425)
0620 20:10:21.488515 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.488520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.489101 (+ 581us) server_negotiation.cc:407] Connection header received
0620 20:10:21.489372 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.489378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.489457 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.489568 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:21.490334 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.491476 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.493317 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.494196 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.494372 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.494483 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.494742 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.494875 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:10:21.495189 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.488308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:42425 (local address 127.4.169.130:46463)
0620 20:10:21.488997 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.489020 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.489166 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.489773 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.489777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.489807 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:21.490170 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.490203 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.491596 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.491600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.493167 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.493181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.494407 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.494414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.494569 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.494593 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.494714 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.494855 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":383,"threads_started":1}
I20260620 20:10:21.495550 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.489448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:44099 (local address 127.4.169.129:39639)
0620 20:10:21.489693 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.489701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.489870 (+ 169us) server_negotiation.cc:407] Connection header received
0620 20:10:21.490068 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.490072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.490127 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.490200 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:21.490720 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.491763 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.493499 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.494644 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.494829 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.494902 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.495179 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.495310 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 20:10:21.495680 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.489129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39639 (local address 127.4.169.130:44099)
0620 20:10:21.489781 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.489798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.489942 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.490247 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.490253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.490281 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:21.490594 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.490602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.491892 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.491898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.493358 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.493370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.494863 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.494869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.495029 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.495051 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.495154 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.495278 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":1,"thread_start_us":372,"threads_started":1}
I20260620 20:10:21.496515 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee24569851df4b23815ecf807439805d" candidate_uuid: "dc74bc6d06de4547bac7512514c33ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7745080856f8400d893c44688e5d00b7" is_pre_election: true
I20260620 20:10:21.496871 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee24569851df4b23815ecf807439805d" candidate_uuid: "dc74bc6d06de4547bac7512514c33ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0" is_pre_election: true
I20260620 20:10:21.497040 12813 raft_consensus.cc:2454] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc74bc6d06de4547bac7512514c33ad5 in term 0.
I20260620 20:10:21.497403 12540 raft_consensus.cc:2454] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc74bc6d06de4547bac7512514c33ad5 in term 0.
I20260620 20:10:21.497804 12606 leader_election.cc:302] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7745080856f8400d893c44688e5d00b7, dc74bc6d06de4547bac7512514c33ad5; no voters:
I20260620 20:10:21.498281 12896 raft_consensus.cc:2781] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:10:21.498452 12896 raft_consensus.cc:482] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:10:21.498581 12896 raft_consensus.cc:3037] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:21.500897 12896 raft_consensus.cc:504] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } }
I20260620 20:10:21.501528 12896 leader_election.cc:288] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [CANDIDATE]: Term 1 election: Requested vote from peers 7745080856f8400d893c44688e5d00b7 (127.4.169.131:42425), dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639)
I20260620 20:10:21.502204 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee24569851df4b23815ecf807439805d" candidate_uuid: "dc74bc6d06de4547bac7512514c33ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7745080856f8400d893c44688e5d00b7"
I20260620 20:10:21.502458 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee24569851df4b23815ecf807439805d" candidate_uuid: "dc74bc6d06de4547bac7512514c33ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0"
I20260620 20:10:21.502485 12813 raft_consensus.cc:3037] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:21.502710 12540 raft_consensus.cc:3037] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:21.504907 12813 raft_consensus.cc:2454] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc74bc6d06de4547bac7512514c33ad5 in term 1.
I20260620 20:10:21.505053 12540 raft_consensus.cc:2454] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc74bc6d06de4547bac7512514c33ad5 in term 1.
I20260620 20:10:21.505412 12606 leader_election.cc:302] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7745080856f8400d893c44688e5d00b7, dc74bc6d06de4547bac7512514c33ad5; no voters:
I20260620 20:10:21.505750 12896 raft_consensus.cc:2781] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:21.506189 12896 raft_consensus.cc:686] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 1 LEADER]: Becoming Leader. State: Replica: dc74bc6d06de4547bac7512514c33ad5, State: Running, Role: LEADER
I20260620 20:10:21.506780 12896 consensus_queue.cc:237] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } }
I20260620 20:10:21.511602 12376 catalog_manager.cc:5495] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130). New cstate: current_term: 1 leader_uuid: "dc74bc6d06de4547bac7512514c33ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } health_report { overall_health: UNKNOWN } } }
I20260620 20:10:21.555168 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.549409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.4.169.130:43333)
0620 20:10:21.549602 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.549607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.549802 (+ 195us) server_negotiation.cc:407] Connection header received
0620 20:10:21.549975 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.549979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.550062 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.550182 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:10:21.550496 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.551180 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.552970 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.553435 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.553531 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.553964 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:10:21.554578 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:10:21.554681 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.554802 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.554867 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:10:21.563422 12540 raft_consensus.cc:1261] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Refusing update from remote peer dc74bc6d06de4547bac7512514c33ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:21.563591 12813 raft_consensus.cc:1261] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Refusing update from remote peer dc74bc6d06de4547bac7512514c33ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:21.564388 12899 consensus_queue.cc:1035] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:21.564725 12896 consensus_queue.cc:1035] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:21.577720 12904 mvcc.cc:205] Tried to move back new op lower bound from 7299015563513896960 to 7299015563295744000. Current Snapshot: MvccSnapshot[applied={T|T < 7299015563513896960}]
I20260620 20:10:21.594376 12378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:21.594658 12378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:21.597718 12378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b: Sending DeleteTablet for 3 replicas of tablet ee24569851df4b23815ecf807439805d
I20260620 20:10:21.598898 12654 tablet_service.cc:1514] Processing DeleteTablet for tablet ee24569851df4b23815ecf807439805d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:21 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:46598
I20260620 20:10:21.598922 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet ee24569851df4b23815ecf807439805d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:21 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:41434
I20260620 20:10:21.599826 12515 tablet_service.cc:1514] Processing DeleteTablet for tablet ee24569851df4b23815ecf807439805d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:21 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:60000
I20260620 20:10:21.600404 12912 tablet_replica.cc:330] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: stopping tablet replica
I20260620 20:10:21.600881 12912 raft_consensus.cc:2229] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:10:21.601468 12912 raft_consensus.cc:2258] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:21.602047 12378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:10:21.603449 12378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:10:21.604374 12912 ts_tablet_manager.cc:1905] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:21.608894 12914 tablet_replica.cc:330] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: stopping tablet replica
I20260620 20:10:21.610239 12914 raft_consensus.cc:2229] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:21.611181 12913 tablet_replica.cc:330] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: stopping tablet replica
I20260620 20:10:21.611761 12914 raft_consensus.cc:2258] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:21.612229 12913 raft_consensus.cc:2229] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:21.612947 12913 raft_consensus.cc:2258] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:21.613480 12912 ts_tablet_manager.cc:1918] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:21.613729 12912 log.cc:1196] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/ee24569851df4b23815ecf807439805d
I20260620 20:10:21.614140 12654 tablet_service.cc:1467] Processing CreateTablet for tablet 6c53faf59b2a43209bf28218bbf6d992 (DEFAULT_TABLE table=test-table [id=488812127d654c91b21485f3d79753c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:21.614378 12912 ts_tablet_manager.cc:1939] T ee24569851df4b23815ecf807439805d P dc74bc6d06de4547bac7512514c33ad5: Deleting consensus metadata
I20260620 20:10:21.614645 12654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c53faf59b2a43209bf28218bbf6d992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:21.614771 12914 ts_tablet_manager.cc:1905] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:21.616487 12364 catalog_manager.cc:4841] TS dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333): tablet ee24569851df4b23815ecf807439805d (table test-table [id=a6ea7e2a850c4f25a2bfc38edb8f7230]) successfully deleted
I20260620 20:10:21.618813 12515 tablet_service.cc:1467] Processing CreateTablet for tablet 6c53faf59b2a43209bf28218bbf6d992 (DEFAULT_TABLE table=test-table [id=488812127d654c91b21485f3d79753c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:21.619355 12515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c53faf59b2a43209bf28218bbf6d992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:21.621706 12890 tablet_bootstrap.cc:492] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: Bootstrap starting.
I20260620 20:10:21.622376 12793 tablet_service.cc:1467] Processing CreateTablet for tablet 6c53faf59b2a43209bf28218bbf6d992 (DEFAULT_TABLE table=test-table [id=488812127d654c91b21485f3d79753c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:21.622843 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c53faf59b2a43209bf28218bbf6d992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:21.623476 12890 tablet_bootstrap.cc:654] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:21.626123 12888 tablet_bootstrap.cc:492] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: Bootstrap starting.
I20260620 20:10:21.627799 12888 tablet_bootstrap.cc:654] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:21.628206 12913 ts_tablet_manager.cc:1905] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:21.634975 12889 tablet_bootstrap.cc:492] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: Bootstrap starting.
I20260620 20:10:21.636754 12889 tablet_bootstrap.cc:654] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:21.640865 12890 tablet_bootstrap.cc:492] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: No bootstrap required, opened a new log
I20260620 20:10:21.641121 12890 ts_tablet_manager.cc:1397] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.009s
I20260620 20:10:21.642093 12890 raft_consensus.cc:348] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.642508 12890 raft_consensus.cc:374] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:21.642669 12890 raft_consensus.cc:729] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc74bc6d06de4547bac7512514c33ad5, State: Initialized, Role: FOLLOWER
I20260620 20:10:21.643018 12890 consensus_queue.cc:260] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.643605 12914 ts_tablet_manager.cc:1918] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:21.643786 12914 log.cc:1196] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/ee24569851df4b23815ecf807439805d
I20260620 20:10:21.644367 12914 ts_tablet_manager.cc:1939] T ee24569851df4b23815ecf807439805d P dda99e976e36442baf918c1d43e6edc0: Deleting consensus metadata
I20260620 20:10:21.646322 12363 catalog_manager.cc:4841] TS dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639): tablet ee24569851df4b23815ecf807439805d (table test-table [id=a6ea7e2a850c4f25a2bfc38edb8f7230]) successfully deleted
I20260620 20:10:21.646507 12913 ts_tablet_manager.cc:1918] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:21.646739 12913 log.cc:1196] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/ee24569851df4b23815ecf807439805d
I20260620 20:10:21.646855 12890 ts_tablet_manager.cc:1428] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260620 20:10:21.647346 12913 ts_tablet_manager.cc:1939] T ee24569851df4b23815ecf807439805d P 7745080856f8400d893c44688e5d00b7: Deleting consensus metadata
I20260620 20:10:21.649220 12889 tablet_bootstrap.cc:492] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: No bootstrap required, opened a new log
I20260620 20:10:21.649283 12363 catalog_manager.cc:4841] TS 7745080856f8400d893c44688e5d00b7 (127.4.169.131:42425): tablet ee24569851df4b23815ecf807439805d (table test-table [id=a6ea7e2a850c4f25a2bfc38edb8f7230]) successfully deleted
I20260620 20:10:21.649431 12889 ts_tablet_manager.cc:1397] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20260620 20:10:21.649966 12888 tablet_bootstrap.cc:492] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: No bootstrap required, opened a new log
I20260620 20:10:21.650192 12888 ts_tablet_manager.cc:1397] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.012s
I20260620 20:10:21.650254 12889 raft_consensus.cc:348] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.650532 12889 raft_consensus.cc:374] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:21.650662 12889 raft_consensus.cc:729] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7745080856f8400d893c44688e5d00b7, State: Initialized, Role: FOLLOWER
I20260620 20:10:21.650951 12889 consensus_queue.cc:260] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.651037 12888 raft_consensus.cc:348] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.651317 12888 raft_consensus.cc:374] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:21.651430 12888 raft_consensus.cc:729] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda99e976e36442baf918c1d43e6edc0, State: Initialized, Role: FOLLOWER
I20260620 20:10:21.651707 12888 consensus_queue.cc:260] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.652087 12889 ts_tablet_manager.cc:1428] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:21.652845 12888 ts_tablet_manager.cc:1428] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 20:10:21.656291 12894 raft_consensus.cc:482] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:10:21.656510 12894 raft_consensus.cc:504] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.658536 12894 leader_election.cc:288] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639), dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333)
I20260620 20:10:21.667297 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.659335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:35961 (local address 127.4.169.130:43333)
0620 20:10:21.659502 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.659510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.660313 (+ 803us) server_negotiation.cc:407] Connection header received
0620 20:10:21.660607 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.660611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.660709 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.660778 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:21.661783 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.662663 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.665406 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.666218 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.666433 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.666527 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.666827 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.666964 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 20:10:21.667424 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.659033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:43333 (local address 127.4.169.131:35961)
0620 20:10:21.660189 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.660206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.660433 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.660957 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.660962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.661002 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:21.661576 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.661583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.662790 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.662798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.665226 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.665234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.666482 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.666489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.666665 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.666690 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.666800 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.666949 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":115,"spinlock_wait_cycles":10240,"thread_start_us":860,"threads_started":1}
I20260620 20:10:21.668007 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.658836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:58839 (local address 127.4.169.129:39639)
0620 20:10:21.659037 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.659047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.660267 (+ 1220us) server_negotiation.cc:407] Connection header received
0620 20:10:21.660605 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.660611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.660689 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.660758 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:21.661765 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.662901 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.665385 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.666744 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.666912 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.666986 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.667589 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.667729 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:10:21.668314 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c53faf59b2a43209bf28218bbf6d992" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc74bc6d06de4547bac7512514c33ad5" is_pre_election: true
I20260620 20:10:21.668704 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.658824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39639 (local address 127.4.169.131:58839)
0620 20:10:21.660155 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.660180 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.660433 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.660958 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.660962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.661000 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:21.661553 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.661580 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.663052 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.663060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.665208 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.665224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.667179 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.667186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.667398 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.667437 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.668193 (+ 756us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.668333 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":66,"spinlock_wait_cycles":39680,"thread_start_us":762,"threads_started":1}
I20260620 20:10:21.668759 12674 raft_consensus.cc:2454] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 0.
I20260620 20:10:21.669440 12746 leader_election.cc:302] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7745080856f8400d893c44688e5d00b7, dc74bc6d06de4547bac7512514c33ad5; no voters:
I20260620 20:10:21.670018 12894 raft_consensus.cc:2781] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:10:21.670140 12894 raft_consensus.cc:482] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:10:21.670246 12894 raft_consensus.cc:3037] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:21.670459 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c53faf59b2a43209bf28218bbf6d992" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0" is_pre_election: true
I20260620 20:10:21.670835 12540 raft_consensus.cc:2454] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 0.
I20260620 20:10:21.673745 12894 raft_consensus.cc:504] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.674518 12894 leader_election.cc:288] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 election: Requested vote from peers dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639), dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333)
I20260620 20:10:21.674981 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c53faf59b2a43209bf28218bbf6d992" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc74bc6d06de4547bac7512514c33ad5"
I20260620 20:10:21.674979 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c53faf59b2a43209bf28218bbf6d992" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0"
I20260620 20:10:21.675196 12674 raft_consensus.cc:3037] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:21.675199 12540 raft_consensus.cc:3037] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:21.677520 12674 raft_consensus.cc:2454] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 1.
I20260620 20:10:21.677994 12746 leader_election.cc:302] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7745080856f8400d893c44688e5d00b7, dc74bc6d06de4547bac7512514c33ad5; no voters:
I20260620 20:10:21.678249 12540 raft_consensus.cc:2454] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 1.
I20260620 20:10:21.678341 12894 raft_consensus.cc:2781] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:21.684351 12894 raft_consensus.cc:686] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 1 LEADER]: Becoming Leader. State: Replica: 7745080856f8400d893c44688e5d00b7, State: Running, Role: LEADER
I20260620 20:10:21.684791 12894 consensus_queue.cc:237] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.688310 12376 catalog_manager.cc:5495] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7745080856f8400d893c44688e5d00b7 (127.4.169.131). New cstate: current_term: 1 leader_uuid: "7745080856f8400d893c44688e5d00b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } health_report { overall_health: HEALTHY } } }
Jun 20 20:10:21 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986218, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.190@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:21.744740 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.728745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42139 (local address 127.0.0.1:53670)
0620 20:10:21.729671 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.729688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.729818 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.730203 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.730207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.730936 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:21.731184 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.731195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.732850 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.732855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.733845 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.733857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.734919 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.734924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.735062 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.735889 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:21.735907 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:21.738450 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:21.742047 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:21.742057 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:21.742064 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:21.742530 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:21.742928 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:21.742932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:21.742935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:21.743094 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:21.743761 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:21.743767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:21.743906 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.744118 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.744353 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":633,"threads_started":1}
I20260620 20:10:21.744966 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.728765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.4.169.190:42139)
0620 20:10:21.729416 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.729423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.729776 (+ 353us) server_negotiation.cc:407] Connection header received
0620 20:10:21.729933 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.729939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.730013 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.730142 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:21.731373 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.732648 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.733986 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.734727 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.734853 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.738661 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:21.738669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:21.738674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:21.738720 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:21.741919 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:21.742633 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:21.742641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:21.742645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:21.742800 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:21.743209 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:21.743214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:21.743218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:21.743598 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:21.743783 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.744227 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.744506 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":484,"thread_start_us":282,"threads_started":1}
I20260620 20:10:21.750134 12376 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:53670:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:10:21.751387 12376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:10:21.757975 12515 tablet_service.cc:1467] Processing CreateTablet for tablet cf6591644d5741a08b8e48da42b0c3f5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=69f88de3d6b0496ebe78a8f1cc291045]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:10:21.758056 12654 tablet_service.cc:1467] Processing CreateTablet for tablet cf6591644d5741a08b8e48da42b0c3f5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=69f88de3d6b0496ebe78a8f1cc291045]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:10:21.758236 12793 tablet_service.cc:1467] Processing CreateTablet for tablet cf6591644d5741a08b8e48da42b0c3f5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=69f88de3d6b0496ebe78a8f1cc291045]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:10:21.758574 12515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6591644d5741a08b8e48da42b0c3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:21.758658 12654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6591644d5741a08b8e48da42b0c3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:21.758740 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6591644d5741a08b8e48da42b0c3f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:21.765961 12890 tablet_bootstrap.cc:492] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: Bootstrap starting.
I20260620 20:10:21.766604 12888 tablet_bootstrap.cc:492] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: Bootstrap starting.
I20260620 20:10:21.768049 12888 tablet_bootstrap.cc:654] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:21.770279 12890 tablet_bootstrap.cc:654] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:21.770924 12889 tablet_bootstrap.cc:492] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: Bootstrap starting.
I20260620 20:10:21.772044 12888 tablet_bootstrap.cc:492] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: No bootstrap required, opened a new log
I20260620 20:10:21.772326 12888 ts_tablet_manager.cc:1397] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260620 20:10:21.772612 12889 tablet_bootstrap.cc:654] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:21.773404 12888 raft_consensus.cc:348] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.773813 12888 raft_consensus.cc:374] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:21.773953 12888 raft_consensus.cc:729] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda99e976e36442baf918c1d43e6edc0, State: Initialized, Role: FOLLOWER
I20260620 20:10:21.774434 12888 consensus_queue.cc:260] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.775110 12895 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.775547 12895 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:10:21.780056 12888 ts_tablet_manager.cc:1428] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260620 20:10:21.784271 12889 tablet_bootstrap.cc:492] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: No bootstrap required, opened a new log
I20260620 20:10:21.784483 12889 ts_tablet_manager.cc:1397] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20260620 20:10:21.785190 12890 tablet_bootstrap.cc:492] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: No bootstrap required, opened a new log
I20260620 20:10:21.785391 12890 ts_tablet_manager.cc:1397] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.010s
I20260620 20:10:21.785331 12889 raft_consensus.cc:348] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.785661 12889 raft_consensus.cc:374] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:21.785741 12889 raft_consensus.cc:729] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7745080856f8400d893c44688e5d00b7, State: Initialized, Role: FOLLOWER
I20260620 20:10:21.785988 12889 consensus_queue.cc:260] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.786556 12894 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.786862 12889 ts_tablet_manager.cc:1428] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 20:10:21.786875 12894 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:10:21.786193 12890 raft_consensus.cc:348] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.787744 12890 raft_consensus.cc:374] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:21.787853 12890 raft_consensus.cc:729] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc74bc6d06de4547bac7512514c33ad5, State: Initialized, Role: FOLLOWER
I20260620 20:10:21.788144 12890 consensus_queue.cc:260] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.788592 12896 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.788883 12896 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:10:21.789307 12890 ts_tablet_manager.cc:1428] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 20:10:21.882541 12896 raft_consensus.cc:482] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:10:21.882772 12896 raft_consensus.cc:504] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.883577 12896 leader_election.cc:288] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639), 7745080856f8400d893c44688e5d00b7 (127.4.169.131:42425)
I20260620 20:10:21.884050 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6591644d5741a08b8e48da42b0c3f5" candidate_uuid: "dc74bc6d06de4547bac7512514c33ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0" is_pre_election: true
I20260620 20:10:21.884313 12540 raft_consensus.cc:2454] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc74bc6d06de4547bac7512514c33ad5 in term 0.
I20260620 20:10:21.884281 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6591644d5741a08b8e48da42b0c3f5" candidate_uuid: "dc74bc6d06de4547bac7512514c33ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7745080856f8400d893c44688e5d00b7" is_pre_election: true
I20260620 20:10:21.884517 12813 raft_consensus.cc:2454] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc74bc6d06de4547bac7512514c33ad5 in term 0.
I20260620 20:10:21.884742 12606 leader_election.cc:302] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc74bc6d06de4547bac7512514c33ad5, dda99e976e36442baf918c1d43e6edc0; no voters:
I20260620 20:10:21.885152 12896 raft_consensus.cc:2781] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:10:21.885277 12896 raft_consensus.cc:482] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:10:21.885375 12896 raft_consensus.cc:3037] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:21.888808 12896 raft_consensus.cc:504] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.889595 12896 leader_election.cc:288] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [CANDIDATE]: Term 1 election: Requested vote from peers dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639), 7745080856f8400d893c44688e5d00b7 (127.4.169.131:42425)
I20260620 20:10:21.890118 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6591644d5741a08b8e48da42b0c3f5" candidate_uuid: "dc74bc6d06de4547bac7512514c33ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0"
I20260620 20:10:21.890206 12813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6591644d5741a08b8e48da42b0c3f5" candidate_uuid: "dc74bc6d06de4547bac7512514c33ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7745080856f8400d893c44688e5d00b7"
I20260620 20:10:21.890375 12540 raft_consensus.cc:3037] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:21.890483 12813 raft_consensus.cc:3037] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:21.892612 12813 raft_consensus.cc:2454] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc74bc6d06de4547bac7512514c33ad5 in term 1.
I20260620 20:10:21.893042 12606 leader_election.cc:302] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7745080856f8400d893c44688e5d00b7, dc74bc6d06de4547bac7512514c33ad5; no voters:
I20260620 20:10:21.893352 12896 raft_consensus.cc:2781] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:21.893471 12540 raft_consensus.cc:2454] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc74bc6d06de4547bac7512514c33ad5 in term 1.
I20260620 20:10:21.893512 12896 raft_consensus.cc:686] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [term 1 LEADER]: Becoming Leader. State: Replica: dc74bc6d06de4547bac7512514c33ad5, State: Running, Role: LEADER
I20260620 20:10:21.893782 12896 consensus_queue.cc:237] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:21.895669 12899 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: TxnStatusTablet state changed. Reason: New leader dc74bc6d06de4547bac7512514c33ad5. Latest consensus state: current_term: 1 leader_uuid: "dc74bc6d06de4547bac7512514c33ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.896108 12899 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:10:21.897133 12933 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:10:21.897395 12933 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:10:21.897260 12378 catalog_manager.cc:5495] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130). New cstate: current_term: 1 leader_uuid: "dc74bc6d06de4547bac7512514c33ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } health_report { overall_health: UNKNOWN } } }
I20260620 20:10:21.918840 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.908698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:43333 (local address 127.0.0.1:46612)
0620 20:10:21.908921 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.908941 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.909059 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.914538 (+ 5479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.914546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.914561 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:10:21.914728 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.914737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.915735 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.915742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.916912 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.916925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.917655 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.917661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.917827 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.917884 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:10:21.918249 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:10:21.918292 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.918377 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.918539 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 20:10:21.918825 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.914012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.4.169.130:43333)
0620 20:10:21.914211 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.914215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.914232 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:10:21.914296 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.914299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.914361 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.914502 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:10:21.914869 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.915610 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.917049 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.917483 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.917627 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.918056 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:10:21.918157 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:10:21.918247 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.918430 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.918507 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:10:21.922274 12540 raft_consensus.cc:1261] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Refusing update from remote peer dc74bc6d06de4547bac7512514c33ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:21.922274 12813 raft_consensus.cc:1261] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Refusing update from remote peer dc74bc6d06de4547bac7512514c33ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:21.922907 12896 consensus_queue.cc:1035] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:21.923185 12899 consensus_queue.cc:1035] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:21.927304 12895 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: TxnStatusTablet state changed. Reason: New leader dc74bc6d06de4547bac7512514c33ad5. Latest consensus state: current_term: 1 leader_uuid: "dc74bc6d06de4547bac7512514c33ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.927608 12895 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:10:21.928982 12909 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc74bc6d06de4547bac7512514c33ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.929316 12909 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:10:21.929484 12902 mvcc.cc:205] Tried to move back new op lower bound from 7299015564987883520 to 7299015564880134144. Current Snapshot: MvccSnapshot[applied={T|T < 7299015564987883520 or (T in {7299015564987883520})}]
I20260620 20:10:21.930442 12903 mvcc.cc:205] Tried to move back new op lower bound from 7299015564987883520 to 7299015564880134144. Current Snapshot: MvccSnapshot[applied={T|T < 7299015564987883520}]
I20260620 20:10:21.931689 12895 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc74bc6d06de4547bac7512514c33ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.931923 12895 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P dda99e976e36442baf918c1d43e6edc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:10:21.946044 12894 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: TxnStatusTablet state changed. Reason: New leader dc74bc6d06de4547bac7512514c33ad5. Latest consensus state: current_term: 1 leader_uuid: "dc74bc6d06de4547bac7512514c33ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.946408 12894 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:10:21.948447 12909 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc74bc6d06de4547bac7512514c33ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.948729 12909 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P dc74bc6d06de4547bac7512514c33ad5: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:10:21.951010 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.936656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.4.169.131:42425)
0620 20:10:21.936826 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.936833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.936854 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:10:21.936970 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.936975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.937095 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.937262 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:10:21.937897 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.939183 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.942982 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.943685 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.943815 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.944599 (+ 784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:10:21.945488 (+ 889us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:10:21.950539 (+ 5051us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.950592 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.950686 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:10:21.952491 12894 tablet_replica.cc:439] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc74bc6d06de4547bac7512514c33ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } } }
I20260620 20:10:21.952808 12894 tablet_replica.cc:442] T cf6591644d5741a08b8e48da42b0c3f5 P 7745080856f8400d893c44688e5d00b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:10:21.968297 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.962593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:52115 (local address 127.4.169.130:43333)
0620 20:10:21.962805 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:21.962815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:21.963264 (+ 449us) server_negotiation.cc:407] Connection header received
0620 20:10:21.963479 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:21.963483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:21.963550 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:21.963635 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:10:21.964113 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.964943 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.966318 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.966851 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.966977 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.967431 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:10:21.967568 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:10:21.967680 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:21.967836 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 20:10:21.967920 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:10:21.968336 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:21.962544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:43333 (local address 127.4.169.131:52115)
0620 20:10:21.963165 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:21.963181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:21.963342 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:21.963663 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:21.963667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:21.963678 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:10:21.963958 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.963966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.965073 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.965079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.966161 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:21.966172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:21.967039 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:21.967046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:21.967190 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:21.967249 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:10:21.967694 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:10:21.967711 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 20:10:21.967810 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:10:21.967951 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":356,"threads_started":1}
I20260620 20:10:21.979135 12540 raft_consensus.cc:1261] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Refusing update from remote peer 7745080856f8400d893c44688e5d00b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:21.979238 12674 raft_consensus.cc:1261] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Refusing update from remote peer 7745080856f8400d893c44688e5d00b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:21.979892 12920 consensus_queue.cc:1035] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:21.980239 12894 consensus_queue.cc:1035] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:22.018564 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:22.011522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:48241 (local address 127.4.169.131:42425)
0620 20:10:22.011701 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:22.011709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:22.012751 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 20:10:22.012974 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:22.012980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:22.013049 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:22.013149 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:10:22.013722 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.014637 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.016454 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.017016 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.017167 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:22.017745 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:10:22.017884 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:10:22.017945 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:22.018149 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 20:10:22.018227 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:10:22.018715 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:22.011566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:42425 (local address 127.4.169.130:48241)
0620 20:10:22.012617 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:22.012642 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:22.012825 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:22.013198 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:22.013204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:22.013218 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:10:22.013499 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:22.013507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.014767 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.014775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:22.016300 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:22.016318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.017211 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.017221 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:22.017442 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:22.017520 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:10:22.018015 (+ 495us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:10:22.018022 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:10:22.018116 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 20:10:22.018268 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":619,"threads_started":1}
I20260620 20:10:22.053098 12378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:22.053361 12378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:22.056994 12378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b: Sending DeleteTablet for 3 replicas of tablet 6c53faf59b2a43209bf28218bbf6d992
I20260620 20:10:22.058354 12515 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c53faf59b2a43209bf28218bbf6d992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:22 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:60000
I20260620 20:10:22.058373 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c53faf59b2a43209bf28218bbf6d992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:22 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:41434
I20260620 20:10:22.058745 12913 tablet_replica.cc:330] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: stopping tablet replica
I20260620 20:10:22.058964 12913 raft_consensus.cc:2229] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:10:22.059500 12913 raft_consensus.cc:2258] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:22.059810 12914 tablet_replica.cc:330] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: stopping tablet replica
I20260620 20:10:22.060031 12914 raft_consensus.cc:2229] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:22.060223 12914 raft_consensus.cc:2258] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:22.061744 12914 ts_tablet_manager.cc:1905] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:22.067080 12913 ts_tablet_manager.cc:1905] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:22.067075 12654 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c53faf59b2a43209bf28218bbf6d992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:22 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:46598
I20260620 20:10:22.067402 12912 tablet_replica.cc:330] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: stopping tablet replica
I20260620 20:10:22.067600 12912 raft_consensus.cc:2229] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:22.067800 12912 raft_consensus.cc:2258] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:22.069063 12912 ts_tablet_manager.cc:1905] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:22.074162 12913 ts_tablet_manager.cc:1918] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:10:22.074316 12913 log.cc:1196] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/6c53faf59b2a43209bf28218bbf6d992
I20260620 20:10:22.074746 12913 ts_tablet_manager.cc:1939] T 6c53faf59b2a43209bf28218bbf6d992 P 7745080856f8400d893c44688e5d00b7: Deleting consensus metadata
I20260620 20:10:22.076040 12363 catalog_manager.cc:4841] TS 7745080856f8400d893c44688e5d00b7 (127.4.169.131:42425): tablet 6c53faf59b2a43209bf28218bbf6d992 (table test-table [id=488812127d654c91b21485f3d79753c5]) successfully deleted
I20260620 20:10:22.081116 12914 ts_tablet_manager.cc:1918] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:10:22.083904 12912 ts_tablet_manager.cc:1918] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:10:22.084209 12914 log.cc:1196] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/6c53faf59b2a43209bf28218bbf6d992
I20260620 20:10:22.084754 12914 ts_tablet_manager.cc:1939] T 6c53faf59b2a43209bf28218bbf6d992 P dda99e976e36442baf918c1d43e6edc0: Deleting consensus metadata
I20260620 20:10:22.084959 12912 log.cc:1196] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/6c53faf59b2a43209bf28218bbf6d992
I20260620 20:10:22.085491 12912 ts_tablet_manager.cc:1939] T 6c53faf59b2a43209bf28218bbf6d992 P dc74bc6d06de4547bac7512514c33ad5: Deleting consensus metadata
I20260620 20:10:22.086779 12363 catalog_manager.cc:4841] TS dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639): tablet 6c53faf59b2a43209bf28218bbf6d992 (table test-table [id=488812127d654c91b21485f3d79753c5]) successfully deleted
I20260620 20:10:22.088069 12364 catalog_manager.cc:4841] TS dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333): tablet 6c53faf59b2a43209bf28218bbf6d992 (table test-table [id=488812127d654c91b21485f3d79753c5]) successfully deleted
Jun 20 20:10:22 dist-test-slave-4bwq krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:10:22 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:22 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:10:22.150933 12928 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:22.136439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.4.169.190:42139)
0620 20:10:22.136652 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:22.136659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:22.137546 (+ 887us) server_negotiation.cc:407] Connection header received
0620 20:10:22.137815 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:22.137822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:22.137913 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:22.138025 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:22.139936 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.141082 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.143014 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.143705 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.143849 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:22.150656 (+ 6807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53684: BlockingRecv error: recv got EOF from 127.0.0.1:53684 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 20:10:22 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM
I20260620 20:10:22.176761 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:22.154326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.4.169.190:42139)
0620 20:10:22.154538 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:22.154545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:22.158292 (+ 3747us) server_negotiation.cc:407] Connection header received
0620 20:10:22.158534 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:22.158541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:22.158630 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:22.158749 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:22.160425 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.161558 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.163478 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.164169 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.164299 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:22.170207 (+ 5908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:22.170215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:22.170219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:22.170274 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:22.173765 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:22.174589 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:22.174595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:22.174599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:22.174784 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:22.175198 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:22.175203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:22.175207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:22.175556 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:22.175816 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:22.176127 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 20:10:22.176392 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 20:10:22.181021 12378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:10:22.181711 12378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:10:22.188920 12515 tablet_service.cc:1467] Processing CreateTablet for tablet c154835be20e447d9ff231112bdbf7e3 (DEFAULT_TABLE table=test-table [id=8a8f963a8fba45c391880ad8d59b1e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:22.189384 12515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c154835be20e447d9ff231112bdbf7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:22.189335 12654 tablet_service.cc:1467] Processing CreateTablet for tablet c154835be20e447d9ff231112bdbf7e3 (DEFAULT_TABLE table=test-table [id=8a8f963a8fba45c391880ad8d59b1e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:22.189786 12654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c154835be20e447d9ff231112bdbf7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:22.193873 12793 tablet_service.cc:1467] Processing CreateTablet for tablet c154835be20e447d9ff231112bdbf7e3 (DEFAULT_TABLE table=test-table [id=8a8f963a8fba45c391880ad8d59b1e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:22.194304 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c154835be20e447d9ff231112bdbf7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:22.196722 12888 tablet_bootstrap.cc:492] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: Bootstrap starting.
I20260620 20:10:22.198335 12888 tablet_bootstrap.cc:654] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:22.206017 12888 tablet_bootstrap.cc:492] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: No bootstrap required, opened a new log
I20260620 20:10:22.206341 12888 ts_tablet_manager.cc:1397] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260620 20:10:22.206899 12890 tablet_bootstrap.cc:492] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: Bootstrap starting.
I20260620 20:10:22.208611 12890 tablet_bootstrap.cc:654] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:22.210935 12890 tablet_bootstrap.cc:492] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: No bootstrap required, opened a new log
I20260620 20:10:22.211149 12890 ts_tablet_manager.cc:1397] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 20:10:22.212028 12890 raft_consensus.cc:348] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.212410 12890 raft_consensus.cc:374] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:22.212550 12890 raft_consensus.cc:729] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc74bc6d06de4547bac7512514c33ad5, State: Initialized, Role: FOLLOWER
I20260620 20:10:22.212882 12890 consensus_queue.cc:260] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.213541 12889 tablet_bootstrap.cc:492] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: Bootstrap starting.
I20260620 20:10:22.215324 12889 tablet_bootstrap.cc:654] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:22.216150 12890 ts_tablet_manager.cc:1428] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 20:10:22.219022 12888 raft_consensus.cc:348] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.219468 12888 raft_consensus.cc:374] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:22.219620 12888 raft_consensus.cc:729] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda99e976e36442baf918c1d43e6edc0, State: Initialized, Role: FOLLOWER
I20260620 20:10:22.220260 12888 consensus_queue.cc:260] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.223656 12888 ts_tablet_manager.cc:1428] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20260620 20:10:22.227877 12889 tablet_bootstrap.cc:492] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: No bootstrap required, opened a new log
I20260620 20:10:22.228116 12889 ts_tablet_manager.cc:1397] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260620 20:10:22.228909 12889 raft_consensus.cc:348] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.229142 12889 raft_consensus.cc:374] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:22.229243 12889 raft_consensus.cc:729] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7745080856f8400d893c44688e5d00b7, State: Initialized, Role: FOLLOWER
I20260620 20:10:22.229499 12889 consensus_queue.cc:260] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.230023 12889 ts_tablet_manager.cc:1428] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 20:10:22.259490 12894 raft_consensus.cc:482] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:10:22.259702 12894 raft_consensus.cc:504] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.260406 12894 leader_election.cc:288] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639), dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333)
I20260620 20:10:22.261057 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c154835be20e447d9ff231112bdbf7e3" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0" is_pre_election: true
I20260620 20:10:22.261091 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c154835be20e447d9ff231112bdbf7e3" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc74bc6d06de4547bac7512514c33ad5" is_pre_election: true
I20260620 20:10:22.261339 12540 raft_consensus.cc:2454] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 0.
I20260620 20:10:22.261344 12674 raft_consensus.cc:2454] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 0.
I20260620 20:10:22.261837 12746 leader_election.cc:302] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7745080856f8400d893c44688e5d00b7, dc74bc6d06de4547bac7512514c33ad5; no voters:
I20260620 20:10:22.262221 12894 raft_consensus.cc:2781] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:10:22.262327 12894 raft_consensus.cc:482] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:10:22.262428 12894 raft_consensus.cc:3037] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:22.265484 12894 raft_consensus.cc:504] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.266659 12674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c154835be20e447d9ff231112bdbf7e3" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc74bc6d06de4547bac7512514c33ad5"
I20260620 20:10:22.266717 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c154835be20e447d9ff231112bdbf7e3" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0"
I20260620 20:10:22.266907 12540 raft_consensus.cc:3037] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:22.266916 12674 raft_consensus.cc:3037] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:22.269906 12540 raft_consensus.cc:2454] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 1.
I20260620 20:10:22.270473 12745 leader_election.cc:302] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7745080856f8400d893c44688e5d00b7, dda99e976e36442baf918c1d43e6edc0; no voters:
I20260620 20:10:22.270788 12920 raft_consensus.cc:2781] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:22.270960 12920 raft_consensus.cc:686] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 1 LEADER]: Becoming Leader. State: Replica: 7745080856f8400d893c44688e5d00b7, State: Running, Role: LEADER
I20260620 20:10:22.271201 12920 consensus_queue.cc:237] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.271636 12674 raft_consensus.cc:2454] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 1.
I20260620 20:10:22.272243 12894 leader_election.cc:288] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 election: Requested vote from peers dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639), dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333)
I20260620 20:10:22.274326 12376 catalog_manager.cc:5495] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7745080856f8400d893c44688e5d00b7 (127.4.169.131). New cstate: current_term: 1 leader_uuid: "7745080856f8400d893c44688e5d00b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } health_report { overall_health: HEALTHY } } }
I20260620 20:10:22.303309 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:22.297875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.4.169.131:42425)
0620 20:10:22.298107 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:22.298113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:22.298178 (+ 65us) server_negotiation.cc:407] Connection header received
0620 20:10:22.298447 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:22.298451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:22.298519 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:22.298624 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:10:22.299001 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.299819 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.301232 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.301690 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.301849 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:22.302427 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:10:22.302547 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:10:22.302651 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:22.302761 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 20:10:22.302845 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 20:10:22.306037 12540 raft_consensus.cc:1261] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Refusing update from remote peer 7745080856f8400d893c44688e5d00b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:22.306046 12674 raft_consensus.cc:1261] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Refusing update from remote peer 7745080856f8400d893c44688e5d00b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:22.306692 12894 consensus_queue.cc:1035] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:22.306986 12942 consensus_queue.cc:1035] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:22.334133 12375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:22.334499 12375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:22.337468 12375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b: Sending DeleteTablet for 3 replicas of tablet c154835be20e447d9ff231112bdbf7e3
I20260620 20:10:22.338222 12515 tablet_service.cc:1514] Processing DeleteTablet for tablet c154835be20e447d9ff231112bdbf7e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:22 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:60000
I20260620 20:10:22.338263 12654 tablet_service.cc:1514] Processing DeleteTablet for tablet c154835be20e447d9ff231112bdbf7e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:22 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:46598
I20260620 20:10:22.338598 12912 tablet_replica.cc:330] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: stopping tablet replica
I20260620 20:10:22.338723 12914 tablet_replica.cc:330] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: stopping tablet replica
I20260620 20:10:22.338805 12912 raft_consensus.cc:2229] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:22.338897 12914 raft_consensus.cc:2229] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:22.339051 12912 raft_consensus.cc:2258] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:22.339112 12914 raft_consensus.cc:2258] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:22.339876 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet c154835be20e447d9ff231112bdbf7e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:22 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:41434
I20260620 20:10:22.340041 12914 ts_tablet_manager.cc:1905] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:22.340148 12913 tablet_replica.cc:330] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: stopping tablet replica
I20260620 20:10:22.340313 12913 raft_consensus.cc:2229] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:10:22.340644 12913 raft_consensus.cc:2258] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:22.341933 12913 ts_tablet_manager.cc:1905] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:22.346541 12914 ts_tablet_manager.cc:1918] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:22.346737 12914 log.cc:1196] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/c154835be20e447d9ff231112bdbf7e3
I20260620 20:10:22.347239 12914 ts_tablet_manager.cc:1939] T c154835be20e447d9ff231112bdbf7e3 P dda99e976e36442baf918c1d43e6edc0: Deleting consensus metadata
I20260620 20:10:22.348306 12363 catalog_manager.cc:4841] TS dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639): tablet c154835be20e447d9ff231112bdbf7e3 (table test-table [id=8a8f963a8fba45c391880ad8d59b1e70]) successfully deleted
I20260620 20:10:22.348109 12375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:10:22.348565 12913 ts_tablet_manager.cc:1918] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:22.348702 12913 log.cc:1196] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/c154835be20e447d9ff231112bdbf7e3
W20260620 20:10:22.348837 12375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:10:22.349138 12913 ts_tablet_manager.cc:1939] T c154835be20e447d9ff231112bdbf7e3 P 7745080856f8400d893c44688e5d00b7: Deleting consensus metadata
I20260620 20:10:22.349942 12363 catalog_manager.cc:4841] TS 7745080856f8400d893c44688e5d00b7 (127.4.169.131:42425): tablet c154835be20e447d9ff231112bdbf7e3 (table test-table [id=8a8f963a8fba45c391880ad8d59b1e70]) successfully deleted
I20260620 20:10:22.350767 12912 ts_tablet_manager.cc:1905] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:22.359097 12515 tablet_service.cc:1467] Processing CreateTablet for tablet b71238a7600a46f9b1647cf6461baec3 (DEFAULT_TABLE table=test-table [id=b1d51ae0128f482d8da8c5159c5bb2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:22.359552 12515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71238a7600a46f9b1647cf6461baec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:22.363610 12654 tablet_service.cc:1467] Processing CreateTablet for tablet b71238a7600a46f9b1647cf6461baec3 (DEFAULT_TABLE table=test-table [id=b1d51ae0128f482d8da8c5159c5bb2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:22.363611 12793 tablet_service.cc:1467] Processing CreateTablet for tablet b71238a7600a46f9b1647cf6461baec3 (DEFAULT_TABLE table=test-table [id=b1d51ae0128f482d8da8c5159c5bb2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:22.364038 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71238a7600a46f9b1647cf6461baec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:22.364053 12654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71238a7600a46f9b1647cf6461baec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:22.365252 12912 ts_tablet_manager.cc:1918] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:22.365435 12912 log.cc:1196] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/c154835be20e447d9ff231112bdbf7e3
I20260620 20:10:22.365981 12912 ts_tablet_manager.cc:1939] T c154835be20e447d9ff231112bdbf7e3 P dc74bc6d06de4547bac7512514c33ad5: Deleting consensus metadata
I20260620 20:10:22.367163 12364 catalog_manager.cc:4841] TS dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333): tablet c154835be20e447d9ff231112bdbf7e3 (table test-table [id=8a8f963a8fba45c391880ad8d59b1e70]) successfully deleted
I20260620 20:10:22.370983 12890 tablet_bootstrap.cc:492] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: Bootstrap starting.
I20260620 20:10:22.372262 12888 tablet_bootstrap.cc:492] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: Bootstrap starting.
I20260620 20:10:22.372684 12890 tablet_bootstrap.cc:654] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:22.373977 12888 tablet_bootstrap.cc:654] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:22.377933 12889 tablet_bootstrap.cc:492] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: Bootstrap starting.
I20260620 20:10:22.380375 12889 tablet_bootstrap.cc:654] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:22.396592 12889 tablet_bootstrap.cc:492] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: No bootstrap required, opened a new log
I20260620 20:10:22.397918 12889 ts_tablet_manager.cc:1397] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20260620 20:10:22.398099 12890 tablet_bootstrap.cc:492] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: No bootstrap required, opened a new log
I20260620 20:10:22.398376 12890 ts_tablet_manager.cc:1397] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.012s
I20260620 20:10:22.398962 12889 raft_consensus.cc:348] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.399248 12889 raft_consensus.cc:374] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:22.399446 12890 raft_consensus.cc:348] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.399804 12890 raft_consensus.cc:374] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:22.399947 12890 raft_consensus.cc:729] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc74bc6d06de4547bac7512514c33ad5, State: Initialized, Role: FOLLOWER
I20260620 20:10:22.400478 12888 tablet_bootstrap.cc:492] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: No bootstrap required, opened a new log
I20260620 20:10:22.400290 12890 consensus_queue.cc:260] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.400650 12888 ts_tablet_manager.cc:1397] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.005s
I20260620 20:10:22.401387 12888 raft_consensus.cc:348] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.401638 12888 raft_consensus.cc:374] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:22.401705 12888 raft_consensus.cc:729] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda99e976e36442baf918c1d43e6edc0, State: Initialized, Role: FOLLOWER
I20260620 20:10:22.402406 12890 ts_tablet_manager.cc:1428] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 20:10:22.403460 12888 consensus_queue.cc:260] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.404057 12888 ts_tablet_manager.cc:1428] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 20:10:22.404233 12889 raft_consensus.cc:729] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7745080856f8400d893c44688e5d00b7, State: Initialized, Role: FOLLOWER
I20260620 20:10:22.404564 12889 consensus_queue.cc:260] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.405612 12889 ts_tablet_manager.cc:1428] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260620 20:10:22.536109 12942 raft_consensus.cc:482] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:10:22.536383 12942 raft_consensus.cc:504] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.537115 12942 leader_election.cc:288] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639), dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333)
I20260620 20:10:22.537835 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71238a7600a46f9b1647cf6461baec3" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0" is_pre_election: true
I20260620 20:10:22.537832 12673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71238a7600a46f9b1647cf6461baec3" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc74bc6d06de4547bac7512514c33ad5" is_pre_election: true
I20260620 20:10:22.538087 12540 raft_consensus.cc:2454] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 0.
I20260620 20:10:22.538094 12673 raft_consensus.cc:2454] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 0.
I20260620 20:10:22.538638 12746 leader_election.cc:302] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7745080856f8400d893c44688e5d00b7, dc74bc6d06de4547bac7512514c33ad5; no voters:
I20260620 20:10:22.538957 12942 raft_consensus.cc:2781] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:10:22.539086 12942 raft_consensus.cc:482] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:10:22.539232 12942 raft_consensus.cc:3037] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:22.541723 12942 raft_consensus.cc:504] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.542784 12673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71238a7600a46f9b1647cf6461baec3" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc74bc6d06de4547bac7512514c33ad5"
I20260620 20:10:22.542766 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71238a7600a46f9b1647cf6461baec3" candidate_uuid: "7745080856f8400d893c44688e5d00b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda99e976e36442baf918c1d43e6edc0"
I20260620 20:10:22.542973 12540 raft_consensus.cc:3037] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:22.542984 12673 raft_consensus.cc:3037] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:22.545930 12673 raft_consensus.cc:2454] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 1.
I20260620 20:10:22.545922 12540 raft_consensus.cc:2454] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7745080856f8400d893c44688e5d00b7 in term 1.
I20260620 20:10:22.546406 12746 leader_election.cc:302] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7745080856f8400d893c44688e5d00b7, dc74bc6d06de4547bac7512514c33ad5; no voters:
I20260620 20:10:22.546715 12920 raft_consensus.cc:2781] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:22.546878 12920 raft_consensus.cc:686] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 1 LEADER]: Becoming Leader. State: Replica: 7745080856f8400d893c44688e5d00b7, State: Running, Role: LEADER
I20260620 20:10:22.546798 12942 leader_election.cc:288] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [CANDIDATE]: Term 1 election: Requested vote from peers dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639), dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333)
I20260620 20:10:22.547183 12920 consensus_queue.cc:237] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } }
I20260620 20:10:22.549883 12378 catalog_manager.cc:5495] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7745080856f8400d893c44688e5d00b7 (127.4.169.131). New cstate: current_term: 1 leader_uuid: "7745080856f8400d893c44688e5d00b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7745080856f8400d893c44688e5d00b7" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 42425 } health_report { overall_health: HEALTHY } } }
I20260620 20:10:22.597335 12540 raft_consensus.cc:1261] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Refusing update from remote peer 7745080856f8400d893c44688e5d00b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:22.598243 12942 consensus_queue.cc:1035] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda99e976e36442baf918c1d43e6edc0" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:22.598472 12673 raft_consensus.cc:1261] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Refusing update from remote peer 7745080856f8400d893c44688e5d00b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:22.599259 12942 consensus_queue.cc:1035] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc74bc6d06de4547bac7512514c33ad5" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:22.688423 12378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:22.688659 12378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:22.691977 12378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51c4eb61385444ca8c1d7016e661539b: Sending DeleteTablet for 3 replicas of tablet b71238a7600a46f9b1647cf6461baec3
I20260620 20:10:22.692791 12654 tablet_service.cc:1514] Processing DeleteTablet for tablet b71238a7600a46f9b1647cf6461baec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:22 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:46598
I20260620 20:10:22.693141 12912 tablet_replica.cc:330] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: stopping tablet replica
I20260620 20:10:22.693117 12515 tablet_service.cc:1514] Processing DeleteTablet for tablet b71238a7600a46f9b1647cf6461baec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:22 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:60000
I20260620 20:10:22.693354 12912 raft_consensus.cc:2229] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:22.693528 12912 raft_consensus.cc:2258] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:22.694509 12914 tablet_replica.cc:330] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: stopping tablet replica
I20260620 20:10:22.694703 12914 raft_consensus.cc:2229] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:22.694847 12914 raft_consensus.cc:2258] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:22.696385 12914 ts_tablet_manager.cc:1905] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:22.696503 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 12449
I20260620 20:10:22.699720 12793 tablet_service.cc:1514] Processing DeleteTablet for tablet b71238a7600a46f9b1647cf6461baec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:22 UTC) from {username='oryx', principal='oryx/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:41434
I20260620 20:10:22.700064 12913 tablet_replica.cc:330] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: stopping tablet replica
I20260620 20:10:22.700270 12913 raft_consensus.cc:2229] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:10:22.700567 12913 raft_consensus.cc:2258] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:22.701704 12913 ts_tablet_manager.cc:1905] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:22.707533 12914 ts_tablet_manager.cc:1918] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:10:22.707809 12914 log.cc:1196] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/b71238a7600a46f9b1647cf6461baec3
I20260620 20:10:22.708302 12914 ts_tablet_manager.cc:1939] T b71238a7600a46f9b1647cf6461baec3 P dda99e976e36442baf918c1d43e6edc0: Deleting consensus metadata
I20260620 20:10:22.708603 12913 ts_tablet_manager.cc:1918] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:10:22.708801 12913 log.cc:1196] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/b71238a7600a46f9b1647cf6461baec3
I20260620 20:10:22.709438 12913 ts_tablet_manager.cc:1939] T b71238a7600a46f9b1647cf6461baec3 P 7745080856f8400d893c44688e5d00b7: Deleting consensus metadata
I20260620 20:10:22.709688 12912 ts_tablet_manager.cc:1905] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 20:10:22 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.129@KRBTEST.COM
I20260620 20:10:22.710809 12363 catalog_manager.cc:4841] TS 7745080856f8400d893c44688e5d00b7 (127.4.169.131:42425): tablet b71238a7600a46f9b1647cf6461baec3 (table test-table [id=b1d51ae0128f482d8da8c5159c5bb2b2]) successfully deleted
I20260620 20:10:22.711483 12363 catalog_manager.cc:4841] TS dda99e976e36442baf918c1d43e6edc0 (127.4.169.129:39639): tablet b71238a7600a46f9b1647cf6461baec3 (table test-table [id=b1d51ae0128f482d8da8c5159c5bb2b2]) successfully deleted
I20260620 20:10:22.714602 12912 ts_tablet_manager.cc:1918] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:10:22.714773 12912 log.cc:1196] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/b71238a7600a46f9b1647cf6461baec3
I20260620 20:10:22.715121 12912 ts_tablet_manager.cc:1939] T b71238a7600a46f9b1647cf6461baec3 P dc74bc6d06de4547bac7512514c33ad5: Deleting consensus metadata
I20260620 20:10:22.716015 12364 catalog_manager.cc:4841] TS dc74bc6d06de4547bac7512514c33ad5 (127.4.169.130:43333): tablet b71238a7600a46f9b1647cf6461baec3 (table test-table [id=b1d51ae0128f482d8da8c5159c5bb2b2]) successfully deleted
I20260620 20:10:22.717191 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:22.699064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60008 (local address 127.4.169.129:39639)
0620 20:10:22.699538 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:22.699573 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:22.699593 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:10:22.699663 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:22.699668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:22.699754 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:22.699868 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:22.701400 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.702447 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.704507 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.705215 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.705376 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:22.711164 (+ 5788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:22.711192 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:22.711208 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:22.711246 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:22.714277 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:22.715019 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:22.715033 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:22.715036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:22.715183 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:22.715546 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:22.715558 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:22.715562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:22.715989 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:22.716320 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:22.716610 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 20:10:22.716843 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":249,"threads_started":1}
I20260620 20:10:22.717896 12580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:22.815738 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 12449
I20260620 20:10:22.833585 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 12589
Jun 20 20:10:22 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.130@KRBTEST.COM
I20260620 20:10:22.847940 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:22.834304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.4.169.130:43333)
0620 20:10:22.834771 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:22.834777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:22.834793 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:10:22.834845 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:22.834849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:22.834923 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:22.835013 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:22.836190 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.836863 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.838062 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:22.838540 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:22.838627 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:22.842783 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:22.842798 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:22.842814 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:22.842847 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:22.845496 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:22.846109 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:22.846116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:22.846119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:22.846249 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:22.846614 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:22.846617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:22.846619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:22.846961 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:22.847174 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:22.847407 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 20:10:22.847567 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":248,"threads_started":1}
I20260620 20:10:22.848574 12719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:22.989984 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 12589
I20260620 20:10:23.017324 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 12728
Jun 20 20:10:23 dist-test-slave-4bwq krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.169.131@KRBTEST.COM
I20260620 20:10:23.033776 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:23.018064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.4.169.131:42425)
0620 20:10:23.018568 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:23.018574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:23.018590 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:10:23.018646 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:23.018649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:23.018721 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:23.018816 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:23.020064 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:23.020770 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:23.021967 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:23.022433 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:23.022566 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:23.026798 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:23.026811 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:23.026827 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:23.026858 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:23.030517 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:23.031414 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:23.031425 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:23.031431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:23.031626 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:23.032095 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:23.032102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:23.032106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:23.032731 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:23.033039 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:23.033291 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 20:10:23.033454 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":254,"threads_started":1}
I20260620 20:10:23.034584 12858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:23.158654 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 12728
I20260620 20:10:23.180639 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 12346
I20260620 20:10:23.181783 12418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:23.295806 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 12346
2026-06-20T20:10:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6768 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:10:23 dist-test-slave-4bwq krb5kdc[12986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:10:23 dist-test-slave-4bwq krb5kdc[12986](info): set up 2 sockets
Jun 20 20:10:23 dist-test-slave-4bwq krb5kdc[12986](info): commencing operation
krb5kdc: starting...
W20260620 20:10:24.397315 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:10:24 dist-test-slave-4bwq krb5kdc[12986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:10:24.624001 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:36843
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:36843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:24.947878 12999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:24.948225 12999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:24.948385 12999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:24.955412 12999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:24.955552 12999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:24.955659 12999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:24.955762 12999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:24.955852 12999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:24.965910 12999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:36843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:36843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:24.967303 12999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:24.968636 12999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:24.980535 13007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:24.980615 13004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:24.980661 13005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:24.981330 12999 server_base.cc:1034] running on GCE node
I20260620 20:10:24.982128 12999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:24.983667 12999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:24.984881 12999 hybrid_clock.cc:648] HybridClock initialized: now 1781986224984816 us; error 57 us; skew 500 ppm
I20260620 20:10:24.985241 12999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T20:10:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1753 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:10:25 dist-test-slave-4bwq krb5kdc[13013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:10:25 dist-test-slave-4bwq krb5kdc[13013](info): set up 2 sockets
Jun 20 20:10:25 dist-test-slave-4bwq krb5kdc[13013](info): commencing operation
krb5kdc: starting...
W20260620 20:10:26.139443 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:10:26 dist-test-slave-4bwq krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:10:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:10:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:10:26.349341 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34607
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:26.670262 13026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:26.670666 13026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:26.670828 13026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:26.678035 13026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:26.678172 13026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:26.678270 13026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:26.678366 13026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:26.678474 13026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:26.688242 13026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:26.689519 13026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:26.690814 13026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:26.702481 13032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:26.702553 13034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:26.702523 13031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:26.703274 13026 server_base.cc:1034] running on GCE node
I20260620 20:10:26.704033 13026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:26.705387 13026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:26.706614 13026 hybrid_clock.cc:648] HybridClock initialized: now 1781986226706549 us; error 66 us; skew 500 ppm
I20260620 20:10:26.706955 13026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:26 dist-test-slave-4bwq krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986226, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:26.710517 13026 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:10:26.711858 13026 webserver.cc:460] Webserver started at http://127.4.169.190:35077/ using document root <none> and password file <none>
I20260620 20:10:26.712518 13026 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:26.712643 13026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:26.712936 13026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:26.715880 13026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "7a5cc82f618946309fc66def2c5a4bdf"
format_stamp: "Formatted at 2026-06-20 20:10:26 on dist-test-slave-4bwq"
server_key: "7e5b29bbc868d60579b770a2009db863"
server_key_iv: "b68df9ebc651feb47574d41e5e8602fd"
server_key_version: "encryptionkey@0"
I20260620 20:10:26.716660 13026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "7a5cc82f618946309fc66def2c5a4bdf"
format_stamp: "Formatted at 2026-06-20 20:10:26 on dist-test-slave-4bwq"
server_key: "7e5b29bbc868d60579b770a2009db863"
server_key_iv: "b68df9ebc651feb47574d41e5e8602fd"
server_key_version: "encryptionkey@0"
I20260620 20:10:26.722180 13026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:10:26.725389 13041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:26.726822 13026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:26.726984 13026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "7a5cc82f618946309fc66def2c5a4bdf"
format_stamp: "Formatted at 2026-06-20 20:10:26 on dist-test-slave-4bwq"
server_key: "7e5b29bbc868d60579b770a2009db863"
server_key_iv: "b68df9ebc651feb47574d41e5e8602fd"
server_key_version: "encryptionkey@0"
I20260620 20:10:26.727197 13026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:26.742601 13026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:26.746601 13026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:26.746891 13026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:26.765846 13026 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:34607
I20260620 20:10:26.765862 13092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:34607 every 8 connection(s)
I20260620 20:10:26.767395 13026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:10:26.771394 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:26.771737 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 13026
I20260620 20:10:26.771963 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:10:26.772787 4774 external_mini_cluster.cc:1428] Setting key 54710391e242fc2f539d5a882ab79249
I20260620 20:10:26.785339 13093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: Bootstrap starting.
Jun 20 20:10:26 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:26.788882 13093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:26.790449 13093 log.cc:824] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:26.793264 13093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: No bootstrap required, opened a new log
I20260620 20:10:26.795975 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:26.774873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.4.169.190:34607)
0620 20:10:26.775648 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:26.775659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:26.775699 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:10:26.777680 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:26.777706 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:26.778087 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:26.778607 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:26.780182 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:26.781427 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:26.783218 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:26.783889 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:26.784027 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:26.788407 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:26.788415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:26.788429 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:26.788467 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:26.791961 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:26.792697 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:26.792709 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:26.792715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:26.792868 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:26.793244 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:26.793257 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:26.793259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:26.793810 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:26.794126 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:26.794550 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 20:10:26.794953 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":368,"threads_started":1}
I20260620 20:10:26.801222 13093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } }
I20260620 20:10:26.801614 13093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:26.801731 13093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5cc82f618946309fc66def2c5a4bdf, State: Initialized, Role: FOLLOWER
I20260620 20:10:26.802271 13093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } }
I20260620 20:10:26.802563 13093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:10:26.802706 13093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:10:26.802877 13093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:26.805392 13093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } }
I20260620 20:10:26.805883 13093 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5cc82f618946309fc66def2c5a4bdf; no voters:
I20260620 20:10:26.806550 13093 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:10:26.806635 13098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:26.807335 13098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 1 LEADER]: Becoming Leader. State: Replica: 7a5cc82f618946309fc66def2c5a4bdf, State: Running, Role: LEADER
I20260620 20:10:26.807930 13098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } }
I20260620 20:10:26.808676 13093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:26.810377 13100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5cc82f618946309fc66def2c5a4bdf. Latest consensus state: current_term: 1 leader_uuid: "7a5cc82f618946309fc66def2c5a4bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } } }
I20260620 20:10:26.810864 13100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:26.810910 13099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5cc82f618946309fc66def2c5a4bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } } }
I20260620 20:10:26.811478 13099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:26.814163 13103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:26.820217 13103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:26.830734 13103 catalog_manager.cc:1347] Generated new cluster ID: c49b9f387fd2452c954c6b54c19dfd22
I20260620 20:10:26.830961 13103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:26.845605 13103 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:10:26.847504 13103 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:26.870150 13103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: Generated new TSK 0
I20260620 20:10:26.871094 13103 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:10:26.929282 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34607
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:27.256143 13120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:27.256397 13120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:27.256500 13120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:27.263445 13120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:27.263551 13120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:27.263664 13120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:10:27.272819 13120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:27.273955 13120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:27.275158 13120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:27.287931 13128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:27.288007 13126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:27.288071 13125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:27.288811 13120 server_base.cc:1034] running on GCE node
I20260620 20:10:27.289311 13120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:27.290114 13120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:27.291307 13120 hybrid_clock.cc:648] HybridClock initialized: now 1781986227291226 us; error 70 us; skew 500 ppm
I20260620 20:10:27.291618 13120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:27 dist-test-slave-4bwq krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986227, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:27.294999 13120 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:10:27.296595 13120 webserver.cc:460] Webserver started at http://127.4.169.129:40823/ using document root <none> and password file <none>
I20260620 20:10:27.297188 13120 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:27.297288 13120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:27.297645 13120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:27.300405 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "e2f05079de524eeeb829c3a382101b89"
format_stamp: "Formatted at 2026-06-20 20:10:27 on dist-test-slave-4bwq"
server_key: "6a68606943d2fd110ed9ed6c7bf442a5"
server_key_iv: "da7eb1efbf90ac696bbe96f9cbd2ebd2"
server_key_version: "encryptionkey@0"
I20260620 20:10:27.301143 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "e2f05079de524eeeb829c3a382101b89"
format_stamp: "Formatted at 2026-06-20 20:10:27 on dist-test-slave-4bwq"
server_key: "6a68606943d2fd110ed9ed6c7bf442a5"
server_key_iv: "da7eb1efbf90ac696bbe96f9cbd2ebd2"
server_key_version: "encryptionkey@0"
I20260620 20:10:27.305765 13120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 20:10:27.309036 13135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:27.310245 13120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:27.310469 13120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "e2f05079de524eeeb829c3a382101b89"
format_stamp: "Formatted at 2026-06-20 20:10:27 on dist-test-slave-4bwq"
server_key: "6a68606943d2fd110ed9ed6c7bf442a5"
server_key_iv: "da7eb1efbf90ac696bbe96f9cbd2ebd2"
server_key_version: "encryptionkey@0"
I20260620 20:10:27.310652 13120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:27.323707 13120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:27.327858 13120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:27.328140 13120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:27.329035 13120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:27.330252 13120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:27.330359 13120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:27.330569 13120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:27.330644 13120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:27.362623 13120 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:41259
I20260620 20:10:27.362705 13247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:41259 every 8 connection(s)
I20260620 20:10:27.363871 13120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:10:27.369879 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 13120
I20260620 20:10:27.370124 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:10:27.370838 4774 external_mini_cluster.cc:1428] Setting key 40424a4369f8d73b24f3c74651de688f
Jun 20 20:10:27 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986227, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:27.384191 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:27.366344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:53107 (local address 127.4.169.190:34607)
0620 20:10:27.366793 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:27.366797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:27.367822 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 20:10:27.369565 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:27.369573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:27.369661 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:27.369837 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:27.371611 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:27.372588 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:27.374092 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:27.374888 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:27.375027 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:27.378181 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:27.378190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:27.378197 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:27.378242 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:27.380995 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:27.381914 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:27.381918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:27.381921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:27.382034 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:27.382533 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:27.382536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:27.382538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:27.382789 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:27.382896 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:27.383766 (+ 870us) server_negotiation.cc:299] Negotiation successful
0620 20:10:27.383931 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":209,"threads_started":1}
I20260620 20:10:27.385664 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:27.366971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:53107)
0620 20:10:27.367661 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:27.367698 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:27.369306 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:27.370092 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:27.370101 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:27.370746 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:27.371402 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:27.371427 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:27.372748 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:27.372752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:27.373945 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:27.373956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:27.375075 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:27.375080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:27.375273 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:27.375924 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:27.375945 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:27.377975 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:27.381137 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:27.381149 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:27.381167 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:27.381800 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:27.382133 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:27.382141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:27.382148 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:27.382419 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:27.382916 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:27.382926 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:27.383375 (+ 449us) client_negotiation.cc:769] Sending connection context
0620 20:10:27.383652 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 20:10:27.384230 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":266,"threads_started":1}
I20260620 20:10:27.387249 13248 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34607
I20260620 20:10:27.387552 13248 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:27.388290 13248 heartbeater.cc:511] Master 127.4.169.190:34607 requested a full tablet report, sending...
I20260620 20:10:27.390476 13058 ts_manager.cc:194] Registered new tserver with Master: e2f05079de524eeeb829c3a382101b89 (127.4.169.129:41259)
I20260620 20:10:27.391862 13058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:53107
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:10:27.420643 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34607
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:27.736673 13254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:27.736996 13254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:27.737178 13254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:27.744462 13254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:27.744638 13254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:27.744845 13254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:10:27.754952 13254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:27.756307 13254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:27.757634 13254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:27.771595 13259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:27.771900 13262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:27.772231 13260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:27.772743 13254 server_base.cc:1034] running on GCE node
I20260620 20:10:27.773464 13254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:27.774688 13254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:27.775928 13254 hybrid_clock.cc:648] HybridClock initialized: now 1781986227775855 us; error 84 us; skew 500 ppm
I20260620 20:10:27.776312 13254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:27 dist-test-slave-4bwq krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986227, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:27.780105 13254 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:10:27.781575 13254 webserver.cc:460] Webserver started at http://127.4.169.130:46327/ using document root <none> and password file <none>
I20260620 20:10:27.782228 13254 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:27.782348 13254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:27.782677 13254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:27.785440 13254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "2b82f99aa0b34d71a62290dadad2c5c6"
format_stamp: "Formatted at 2026-06-20 20:10:27 on dist-test-slave-4bwq"
server_key: "cf486147c4fdd03a6240f653d8648643"
server_key_iv: "bb94352518294e461c777542d3003f7f"
server_key_version: "encryptionkey@0"
I20260620 20:10:27.786188 13254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "2b82f99aa0b34d71a62290dadad2c5c6"
format_stamp: "Formatted at 2026-06-20 20:10:27 on dist-test-slave-4bwq"
server_key: "cf486147c4fdd03a6240f653d8648643"
server_key_iv: "bb94352518294e461c777542d3003f7f"
server_key_version: "encryptionkey@0"
I20260620 20:10:27.791270 13254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260620 20:10:27.794623 13269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:27.795912 13254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:10:27.796028 13254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "2b82f99aa0b34d71a62290dadad2c5c6"
format_stamp: "Formatted at 2026-06-20 20:10:27 on dist-test-slave-4bwq"
server_key: "cf486147c4fdd03a6240f653d8648643"
server_key_iv: "bb94352518294e461c777542d3003f7f"
server_key_version: "encryptionkey@0"
I20260620 20:10:27.796177 13254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:27.817515 13254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:27.821834 13254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:27.822069 13254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:27.822992 13254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:27.824193 13254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:27.824266 13254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:27.824347 13254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:27.824389 13254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:27.857062 13254 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:46033
I20260620 20:10:27.857169 13381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:46033 every 8 connection(s)
I20260620 20:10:27.858315 13254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:10:27.860054 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 13254
I20260620 20:10:27.860255 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:10:27.860718 4774 external_mini_cluster.cc:1428] Setting key e5624b6deed7fa10486adc79f24eac69
Jun 20 20:10:27 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986227, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:27.882320 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:27.861158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:53411 (local address 127.4.169.190:34607)
0620 20:10:27.861356 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:27.861363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:27.862695 (+ 1332us) server_negotiation.cc:407] Connection header received
0620 20:10:27.865528 (+ 2833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:27.865536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:27.865612 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:27.865717 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:27.867790 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:27.868481 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:27.871063 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:27.871791 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:27.871936 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:27.875522 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:27.875531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:27.875540 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:27.875583 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:27.879047 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:27.879738 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:27.879746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:27.879752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:27.879906 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:27.880347 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:27.880355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:27.880362 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:27.880738 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:27.880895 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:27.881623 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 20:10:27.881853 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:10:27.883044 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:27.861663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:53411)
0620 20:10:27.862511 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:27.862580 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:27.865236 (+ 2656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:27.865985 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:27.865999 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:27.866775 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:27.867593 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:27.867624 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:27.868643 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:27.868651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:27.870893 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:27.870912 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:27.872010 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:27.872014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:27.872229 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:27.872921 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:27.872947 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:27.875277 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:27.879184 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:27.879191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:27.879203 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:27.879603 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:27.880022 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:27.880025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:27.880027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:27.880215 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:27.880861 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:27.880867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:27.881175 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 20:10:27.881492 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 20:10:27.881872 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":293,"threads_started":1}
I20260620 20:10:27.884491 13382 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34607
I20260620 20:10:27.884785 13382 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:10:27.885507 13382 heartbeater.cc:511] Master 127.4.169.190:34607 requested a full tablet report, sending...
I20260620 20:10:27.887025 13058 ts_manager.cc:194] Registered new tserver with Master: 2b82f99aa0b34d71a62290dadad2c5c6 (127.4.169.130:46033)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
I20260620 20:10:27.888270 13058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:53411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:10:27.911535 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34607
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:28.228847 13387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:28.229159 13387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:28.229326 13387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:28.236554 13387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:28.236696 13387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:28.236877 13387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:10:28.246481 13387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:28.247784 13387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:28.249071 13387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:28.261780 13392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:28.261883 13393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:28.261852 13395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:28.262636 13387 server_base.cc:1034] running on GCE node
I20260620 20:10:28.263154 13387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:28.264024 13387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:28.265241 13387 hybrid_clock.cc:648] HybridClock initialized: now 1781986228265199 us; error 32 us; skew 500 ppm
I20260620 20:10:28.265535 13387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:28 dist-test-slave-4bwq krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986228, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:28.268554 13387 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:10:28.269867 13387 webserver.cc:460] Webserver started at http://127.4.169.131:44853/ using document root <none> and password file <none>
I20260620 20:10:28.270669 13387 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:28.270800 13387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:28.271104 13387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:28.273829 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "4e0db79699624c65901f13df7110611e"
format_stamp: "Formatted at 2026-06-20 20:10:28 on dist-test-slave-4bwq"
server_key: "665ab5e054e4978771365ba9777e2a73"
server_key_iv: "0c457f5175b814cf70453ad8bd7e503e"
server_key_version: "encryptionkey@0"
I20260620 20:10:28.274638 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "4e0db79699624c65901f13df7110611e"
format_stamp: "Formatted at 2026-06-20 20:10:28 on dist-test-slave-4bwq"
server_key: "665ab5e054e4978771365ba9777e2a73"
server_key_iv: "0c457f5175b814cf70453ad8bd7e503e"
server_key_version: "encryptionkey@0"
I20260620 20:10:28.279505 13387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:10:28.282867 13402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:28.284269 13387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:10:28.284439 13387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "4e0db79699624c65901f13df7110611e"
format_stamp: "Formatted at 2026-06-20 20:10:28 on dist-test-slave-4bwq"
server_key: "665ab5e054e4978771365ba9777e2a73"
server_key_iv: "0c457f5175b814cf70453ad8bd7e503e"
server_key_version: "encryptionkey@0"
I20260620 20:10:28.284638 13387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:28.296684 13387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:28.300987 13387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:28.301261 13387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:28.302125 13387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:28.303442 13387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:28.303543 13387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:28.303683 13387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:28.303752 13387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:28.336310 13387 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:43587
I20260620 20:10:28.336349 13514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:43587 every 8 connection(s)
I20260620 20:10:28.337550 13387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:10:28.341740 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 13387
I20260620 20:10:28.341943 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:10:28.342490 4774 external_mini_cluster.cc:1428] Setting key 4c709fca7ecebdad5b1c71835d540059
Jun 20 20:10:28 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986228, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:28.356690 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:28.339983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:43675 (local address 127.4.169.190:34607)
0620 20:10:28.340188 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:28.340195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:28.341364 (+ 1169us) server_negotiation.cc:407] Connection header received
0620 20:10:28.343763 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:28.343768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:28.343850 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:28.343956 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:28.345637 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.346369 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.347923 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.348321 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.348449 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:28.351626 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:28.351631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:28.351634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:28.351665 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:28.353992 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:28.354674 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:28.354678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:28.354680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:28.354812 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:28.355151 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:28.355154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:28.355156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:28.355395 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:28.355498 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:28.356149 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 20:10:28.356366 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:10:28.357661 13517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:28.340530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:43675)
0620 20:10:28.341201 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:28.341238 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:28.343455 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:28.344179 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:28.344187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:28.344857 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:28.345484 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:28.345500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.346566 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.346570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:28.347773 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:28.347789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.348531 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.348539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:28.348771 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:28.349362 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:28.349382 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:28.351448 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:28.354108 (+ 2660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:28.354115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:28.354126 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:28.354575 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:28.354900 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:28.354903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:28.354905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:28.355061 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:28.355490 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:28.355496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:28.355767 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 20:10:28.356017 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 20:10:28.356487 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":276,"threads_started":1}
I20260620 20:10:28.358901 13515 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34607
I20260620 20:10:28.359200 13515 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:28.359892 13515 heartbeater.cc:511] Master 127.4.169.190:34607 requested a full tablet report, sending...
I20260620 20:10:28.361155 13058 ts_manager.cc:194] Registered new tserver with Master: 4e0db79699624c65901f13df7110611e (127.4.169.131:43587)
I20260620 20:10:28.361999 13058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:43675
I20260620 20:10:28.369235 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.129@KRBTEST.COM" created.
I20260620 20:10:28.395107 13248 heartbeater.cc:503] Master 127.4.169.190:34607 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab.
Entry for principal oryx/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab.
WARNING: no policy specified for oryx/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab.
Entry for principal oryx/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab.
WARNING: no policy specified for oryx/127.4.169.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab.
Entry for principal oryx/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab.
I20260620 20:10:28.495082 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13120
Jun 20 20:10:28 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:10:28.510537 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:28.496135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.4.169.129:41259)
0620 20:10:28.496809 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:28.496816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:28.496842 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:10:28.496902 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:28.496908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:28.497120 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:28.497258 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:28.498539 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.499519 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.500843 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.501393 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.501491 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:28.505318 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:28.505329 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:28.505343 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:28.505378 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:28.508015 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:28.508667 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:28.508674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:28.508676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:28.508807 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:28.509134 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:28.509138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:28.509140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:28.509500 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:28.509760 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:28.509978 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 20:10:28.510160 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":354,"threads_started":1}
I20260620 20:10:28.511751 13243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:28.592794 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 13120
I20260620 20:10:28.609515 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13254
Jun 20 20:10:28 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:10:28.624845 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:28.610342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.4.169.130:46033)
0620 20:10:28.610897 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:28.610904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:28.610929 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:10:28.610994 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:28.611000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:28.611213 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:28.611348 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:28.612481 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.613471 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.614845 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.615488 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.615593 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:28.619719 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:28.619730 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:28.619744 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:28.619776 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:28.622501 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:28.623131 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:28.623138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:28.623140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:28.623258 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:28.623546 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:28.623551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:28.623554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:28.623859 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:28.624122 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:28.624274 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 20:10:28.624453 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":314,"threads_started":1}
I20260620 20:10:28.626011 13377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:28.706527 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 13254
I20260620 20:10:28.723730 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13387
Jun 20 20:10:28 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:10:28.739626 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:28.724765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.4.169.131:43587)
0620 20:10:28.725345 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:28.725352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:28.725380 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:10:28.725435 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:28.725442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:28.725647 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:28.725826 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:28.727118 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.728117 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.729347 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:28.730011 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:28.730111 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:28.734338 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:28.734350 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:28.734364 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:28.734525 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:28.737136 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:28.737743 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:28.737749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:28.737752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:28.737871 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:28.738166 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:28.738169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:28.738170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:28.738588 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:28.738844 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:28.739042 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 20:10:28.739234 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":301,"threads_started":1}
I20260620 20:10:28.740885 13510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:28.823753 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 13387
I20260620 20:10:28.839469 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13026
I20260620 20:10:28.840536 13088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:28.941965 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 13026
I20260620 20:10:28.961897 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34607
--webserver_interface=127.4.169.190
--webserver_port=35077
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:29.284842 13532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:29.285182 13532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:29.285364 13532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:29.292711 13532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:29.292802 13532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:29.292847 13532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:29.292919 13532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:29.293000 13532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:29.302800 13532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=35077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:29.304064 13532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:29.305339 13532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:29.316802 13537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:29.316824 13538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:29.316764 13540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:29.317893 13532 server_base.cc:1034] running on GCE node
I20260620 20:10:29.318792 13532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:29.320151 13532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:29.321375 13532 hybrid_clock.cc:648] HybridClock initialized: now 1781986229321326 us; error 44 us; skew 500 ppm
I20260620 20:10:29.321748 13532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:29 dist-test-slave-4bwq krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986229, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:29.324716 13532 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:10:29.326149 13532 webserver.cc:460] Webserver started at http://127.4.169.190:35077/ using document root <none> and password file <none>
I20260620 20:10:29.326891 13532 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:29.327052 13532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:29.332286 13532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:10:29.335031 13547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:29.336350 13532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:29.336521 13532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "7a5cc82f618946309fc66def2c5a4bdf"
format_stamp: "Formatted at 2026-06-20 20:10:26 on dist-test-slave-4bwq"
server_key: "7e5b29bbc868d60579b770a2009db863"
server_key_iv: "b68df9ebc651feb47574d41e5e8602fd"
server_key_version: "encryptionkey@0"
I20260620 20:10:29.337462 13532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:29.351989 13532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:29.356184 13532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:29.356467 13532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:29.373976 13532 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:34607
I20260620 20:10:29.373997 13598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:34607 every 8 connection(s)
I20260620 20:10:29.375391 13532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:10:29.381826 13599 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:10:29.383086 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 13532
I20260620 20:10:29.384032 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:41259
--local_ip_for_outbound_sockets=127.4.169.129
--tserver_master_addrs=127.4.169.190:34607
--webserver_port=40823
--webserver_interface=127.4.169.129
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.169.129
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:10:29.386291 13599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: Bootstrap starting.
I20260620 20:10:29.406495 13599 log.cc:824] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:29.415184 13599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:10:29.415717 13599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: Bootstrap complete.
I20260620 20:10:29.424726 13599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } }
I20260620 20:10:29.425542 13599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5cc82f618946309fc66def2c5a4bdf, State: Initialized, Role: FOLLOWER
I20260620 20:10:29.426134 13599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } }
I20260620 20:10:29.426371 13599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:10:29.426515 13599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:10:29.426715 13599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:10:29.430344 13599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } }
I20260620 20:10:29.430972 13599 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5cc82f618946309fc66def2c5a4bdf; no voters:
I20260620 20:10:29.431643 13599 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:10:29.431782 13603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:10:29.433125 13603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [term 2 LEADER]: Becoming Leader. State: Replica: 7a5cc82f618946309fc66def2c5a4bdf, State: Running, Role: LEADER
I20260620 20:10:29.433743 13603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } }
I20260620 20:10:29.433884 13599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:29.436419 13605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5cc82f618946309fc66def2c5a4bdf. Latest consensus state: current_term: 2 leader_uuid: "7a5cc82f618946309fc66def2c5a4bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } } }
I20260620 20:10:29.436828 13605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:29.437431 13604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7a5cc82f618946309fc66def2c5a4bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5cc82f618946309fc66def2c5a4bdf" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34607 } } }
I20260620 20:10:29.437745 13604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:29.446767 13611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:29.452518 13611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:29.453941 13611 catalog_manager.cc:1259] Loaded cluster ID: c49b9f387fd2452c954c6b54c19dfd22
I20260620 20:10:29.454041 13611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:29.457917 13611 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:29.459434 13611 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7a5cc82f618946309fc66def2c5a4bdf: Loaded TSK: 0
I20260620 20:10:29.461020 13611 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:10:29.728711 13601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:29.729022 13601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:29.729192 13601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:29.736935 13601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:29.737089 13601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:29.737277 13601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:10:29.747743 13601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.129.keytab
--principal=oryx/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:41259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=40823
--tserver_master_addrs=127.4.169.190:34607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:29.749156 13601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:29.750629 13601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:29.765849 13626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:29.766803 13625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:29.768158 13628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:29.769055 13601 server_base.cc:1034] running on GCE node
I20260620 20:10:29.769856 13601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:29.771066 13601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:29.772408 13601 hybrid_clock.cc:648] HybridClock initialized: now 1781986229772355 us; error 42 us; skew 500 ppm
I20260620 20:10:29.772759 13601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:29 dist-test-slave-4bwq krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986229, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:29.778956 13601 init.cc:377] Logged in from keytab as oryx/127.4.169.129@KRBTEST.COM (short username oryx)
I20260620 20:10:29.780532 13601 webserver.cc:460] Webserver started at http://127.4.169.129:40823/ using document root <none> and password file <none>
I20260620 20:10:29.781222 13601 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:29.781376 13601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:29.786764 13601 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:29.789371 13635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:29.790804 13601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:29.790963 13601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "e2f05079de524eeeb829c3a382101b89"
format_stamp: "Formatted at 2026-06-20 20:10:27 on dist-test-slave-4bwq"
server_key: "6a68606943d2fd110ed9ed6c7bf442a5"
server_key_iv: "da7eb1efbf90ac696bbe96f9cbd2ebd2"
server_key_version: "encryptionkey@0"
I20260620 20:10:29.791862 13601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:29.810786 13601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:29.814985 13601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:29.815263 13601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:29.816184 13601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:29.817430 13601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:29.817538 13601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:29.817680 13601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:29.817762 13601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:29.850526 13601 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:41259
I20260620 20:10:29.850576 13747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:41259 every 8 connection(s)
I20260620 20:10:29.851897 13601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:10:29.858949 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 13601
I20260620 20:10:29.859568 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:46033
--local_ip_for_outbound_sockets=127.4.169.130
--tserver_master_addrs=127.4.169.190:34607
--webserver_port=46327
--webserver_interface=127.4.169.130
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.169.130
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:10:29 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:29 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:10:30.182984 13752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:30.183259 13752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:30.183408 13752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:30.190639 13752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:30.190744 13752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:30.190948 13752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:10:30.200807 13752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.130.keytab
--principal=oryx/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:46033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=46327
--tserver_master_addrs=127.4.169.190:34607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:30.202090 13752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:30.203362 13752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:30.221099 13758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:30.221128 13759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:30.221735 13752 server_base.cc:1034] running on GCE node
W20260620 20:10:30.221869 13761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:30.222666 13752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:30.223671 13752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:30.224928 13752 hybrid_clock.cc:648] HybridClock initialized: now 1781986230224870 us; error 55 us; skew 500 ppm
I20260620 20:10:30.225234 13752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:30 dist-test-slave-4bwq krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986230, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:30.228686 13752 init.cc:377] Logged in from keytab as oryx/127.4.169.130@KRBTEST.COM (short username oryx)
I20260620 20:10:30.230525 13752 webserver.cc:460] Webserver started at http://127.4.169.130:46327/ using document root <none> and password file <none>
I20260620 20:10:30.231451 13752 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:30.231601 13752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:30.237599 13752 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 20:10:30.240697 13768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:30.242532 13752 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 20:10:30.242698 13752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "2b82f99aa0b34d71a62290dadad2c5c6"
format_stamp: "Formatted at 2026-06-20 20:10:27 on dist-test-slave-4bwq"
server_key: "cf486147c4fdd03a6240f653d8648643"
server_key_iv: "bb94352518294e461c777542d3003f7f"
server_key_version: "encryptionkey@0"
I20260620 20:10:30.243618 13752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:30.272513 13752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:30.276841 13752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:30.277112 13752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:30.278153 13752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:30.279609 13752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:30.279734 13752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:30.279875 13752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:30.279961 13752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:30.319600 13752 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:46033
I20260620 20:10:30.319648 13880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:46033 every 8 connection(s)
I20260620 20:10:30.321050 13752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:10:30.323068 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 13752
I20260620 20:10:30.323683 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:43587
--local_ip_for_outbound_sockets=127.4.169.131
--tserver_master_addrs=127.4.169.190:34607
--webserver_port=44853
--webserver_interface=127.4.169.131
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.169.131
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:10:30 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:30 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:10:30.677165 13885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:30.677484 13885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:30.677657 13885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:30.684913 13885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:30.685067 13885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:30.685268 13885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:10:30.695677 13885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/krb5kdc/oryx_127.4.169.131.keytab
--principal=oryx/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:43587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=44853
--tserver_master_addrs=127.4.169.190:34607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:30.697479 13885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:30.699016 13885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:30.716741 13891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:30.718426 13892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:30.724828 13895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:30.725852 13885 server_base.cc:1034] running on GCE node
I20260620 20:10:30.726588 13885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:30.727725 13885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:30.728971 13885 hybrid_clock.cc:648] HybridClock initialized: now 1781986230728913 us; error 63 us; skew 500 ppm
I20260620 20:10:30.729310 13885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:30 dist-test-slave-4bwq krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986230, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:30.732517 13885 init.cc:377] Logged in from keytab as oryx/127.4.169.131@KRBTEST.COM (short username oryx)
I20260620 20:10:30.738746 13885 webserver.cc:460] Webserver started at http://127.4.169.131:44853/ using document root <none> and password file <none>
I20260620 20:10:30.739562 13885 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:30.740229 13885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:30.746114 13885 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 20:10:30.749135 13901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:30.750662 13885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:30.750840 13885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "4e0db79699624c65901f13df7110611e"
format_stamp: "Formatted at 2026-06-20 20:10:28 on dist-test-slave-4bwq"
server_key: "665ab5e054e4978771365ba9777e2a73"
server_key_iv: "0c457f5175b814cf70453ad8bd7e503e"
server_key_version: "encryptionkey@0"
I20260620 20:10:30.751788 13885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:30.780697 13885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:30.785116 13885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:30.785400 13885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:30.786466 13885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:30.787925 13885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:30.788045 13885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:30.788182 13885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:30.788276 13885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:30.836395 13885 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:43587
I20260620 20:10:30.836439 14013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:43587 every 8 connection(s)
I20260620 20:10:30.837852 13885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:10:30.840341 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 13885
Jun 20 20:10:30 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:30 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:30.880596 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:30.842202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.4.169.190:34607)
0620 20:10:30.845093 (+ 2891us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:30.845099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:30.845117 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:10:30.845184 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:30.845191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:30.845281 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:30.845387 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:30.848649 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.860175 (+ 11526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.862948 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.863616 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.863743 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:30.870598 (+ 6855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:30.870609 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:30.870635 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:30.870700 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:30.875022 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:30.875852 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:30.875864 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:30.875874 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:30.876093 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:30.876559 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:30.876565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:30.876569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:30.877230 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:30.877548 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:30.879020 (+ 1472us) server_negotiation.cc:299] Negotiation successful
0620 20:10:30.879471 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2790,"thread_start_us":2672,"threads_started":1}
W20260620 20:10:31.412606 13751 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.546s user 0.001s sys 0.012s
W20260620 20:10:31.414840 13750 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:29.854804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:35241 (local address 127.4.169.190:34607)
0620 20:10:29.855466 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:29.855477 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:29.855890 (+ 413us) server_negotiation.cc:407] Connection header received
0620 20:10:29.857867 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:29.857879 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:29.858187 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:29.858625 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:29.860186 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:29.862199 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:29.863753 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:29.864883 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:29.865069 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.414342 (+1549273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:35241: BlockingRecv error: recv got EOF from 127.4.169.129:35241 (error 108)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":259,"threads_started":1}
I20260620 20:10:31.416509 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:29.854896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:35241)
0620 20:10:29.855705 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:29.855768 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:29.857407 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:29.858703 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:29.858710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:29.859281 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:29.859964 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:29.859998 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:29.862366 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:29.862370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:29.863608 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:29.863619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:29.865199 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:29.865204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:29.865392 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:29.866054 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:29.866075 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:31.414688 (+1548613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":353,"threads_started":1}
W20260620 20:10:31.418109 13748 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:34607 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.169.190:34607: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Jun 20 20:10:31 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:31 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:10:31.436765 13750 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:31.419621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:57441 (local address 127.4.169.190:34607)
0620 20:10:31.420184 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:31.420193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:31.420219 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:10:31.420291 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:31.420299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:31.420406 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:31.420543 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:31.421963 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.423696 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.426596 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.427590 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.427780 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.436332 (+ 8552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:57441: BlockingRecv error: recv got EOF from 127.4.169.129:57441 (error 108)
Metrics: {"server-negotiator.queue_time_us":380}
I20260620 20:10:31.437538 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:31.419270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:57441)
0620 20:10:31.419575 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:31.419605 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:31.419807 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:31.420670 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:31.420679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:31.421380 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:31.421715 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:31.421732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.423919 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.423928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:31.426351 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:31.426375 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.427977 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.427988 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:31.428302 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.429302 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:31.429339 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:31.437094 (+ 7755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
Jun 20 20:10:31 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:31 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:31.456873 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:31.438736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:40413)
0620 20:10:31.439080 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:31.439107 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:31.439278 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:31.439802 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:31.439810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:31.440441 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:31.440785 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:31.440802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.443184 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.443193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:31.445560 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:31.445585 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.448280 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.448292 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:31.448598 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.449608 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:31.449645 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:31.456398 (+ 6753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188}
W20260620 20:10:31.457275 13750 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:31.438840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:40413 (local address 127.4.169.190:34607)
0620 20:10:31.439094 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:31.439104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:31.439174 (+ 70us) server_negotiation.cc:407] Connection header received
0620 20:10:31.439472 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:31.439481 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:31.439592 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:31.439741 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:31.440982 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.442969 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.445817 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.447649 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.447821 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.456779 (+ 8958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:40413: BlockingRecv error: recv got EOF from 127.4.169.129:40413 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 20:10:31.457644 13748 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:10:31.808730 13884 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.458s user 0.007s sys 0.017s
W20260620 20:10:31.810480 13883 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:30.324297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:36135 (local address 127.4.169.190:34607)
0620 20:10:30.338824 (+ 14527us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:30.338832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:30.339197 (+ 365us) server_negotiation.cc:407] Connection header received
0620 20:10:30.341226 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:30.341237 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:30.341370 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:30.341535 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:30.343719 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.345409 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.348093 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.349076 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.349278 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.810174 (+1460896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:36135: BlockingRecv error: recv got EOF from 127.4.169.130:36135 (error 108)
Metrics: {"server-negotiator.queue_time_us":14372,"thread_start_us":14257,"threads_started":1}
I20260620 20:10:31.811718 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:30.325253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:36135)
0620 20:10:30.338995 (+ 13742us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:30.339052 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:30.340902 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:30.341825 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:30.341839 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:30.342667 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:30.343489 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:30.343518 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.345611 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.345619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:30.347907 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:30.347924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.349341 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.349351 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:30.349686 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:30.350660 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:30.350704 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:31.810032 (+1459328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13423,"thread_start_us":13163,"threads_started":1}
W20260620 20:10:31.812810 13881 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:34607 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.169.190:34607: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Jun 20 20:10:31 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:31 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:31.822624 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:31.813425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:50603)
0620 20:10:31.813714 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:31.813743 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:31.813911 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:31.814312 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:31.814319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:31.814846 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:31.815061 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:31.815070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.816297 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.816303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:31.817248 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:31.817259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.817932 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.817938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:31.818110 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.818776 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:31.818794 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:31.822293 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 20:10:31.822741 13883 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:31.813658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:50603 (local address 127.4.169.190:34607)
0620 20:10:31.813811 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:31.813818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:31.813838 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:10:31.814089 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:31.814094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:31.814172 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:31.814271 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:31.815199 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.816162 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.817391 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.817768 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.817903 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.822508 (+ 4605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:50603: BlockingRecv error: recv got EOF from 127.4.169.130:50603 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 20:10:31 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:31 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:31.836057 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:31.823352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:35017)
0620 20:10:31.823502 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:31.823527 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:31.823643 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:31.824287 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:31.824290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:31.824550 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:31.824703 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:31.824710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.826783 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.826792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:31.829132 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:31.829154 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.829877 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.829886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:31.830189 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.830933 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:31.830967 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:31.835675 (+ 4708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:10:31.836285 13883 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:31.823652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:35017 (local address 127.4.169.190:34607)
0620 20:10:31.823882 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:31.823891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:31.823945 (+ 54us) server_negotiation.cc:407] Connection header received
0620 20:10:31.824017 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:31.824025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:31.824143 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:31.824258 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:31.824842 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.826544 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.829299 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:31.829707 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:31.829801 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:31.835875 (+ 6074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:35017: BlockingRecv error: recv got EOF from 127.4.169.130:35017 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 20:10:31.842984 13881 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:10:32.012713 14018 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.141s user 0.000s sys 0.015s
W20260620 20:10:32.013767 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:30.847806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:37077 (local address 127.4.169.190:34607)
0620 20:10:30.858757 (+ 10951us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:30.858762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:30.859135 (+ 373us) server_negotiation.cc:407] Connection header received
0620 20:10:30.863144 (+ 4009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:30.863149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:30.863232 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:30.863326 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:30.866026 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.867251 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.869789 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.870463 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.870554 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:32.013530 (+1142976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:37077: BlockingRecv error: recv got EOF from 127.4.169.131:37077 (error 108)
Metrics: {"server-negotiator.queue_time_us":10817,"spinlock_wait_cycles":14080,"thread_start_us":11491,"threads_started":1}
I20260620 20:10:32.014309 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:30.847158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:37077)
0620 20:10:30.858919 (+ 11761us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:30.859008 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:30.862746 (+ 3738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:30.863809 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:30.863824 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:30.864798 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:30.865799 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:30.865830 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.867427 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.867432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:30.869613 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:30.869630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:30.870967 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:30.870974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:30.871285 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:30.871953 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:30.871990 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:32.013366 (+1141376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11368,"thread_start_us":11125,"threads_started":1}
W20260620 20:10:32.015125 14014 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:34607 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.169.190:34607: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Jun 20 20:10:32 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:32 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:32.024068 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:32.015753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:53523)
0620 20:10:32.015948 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:32.015970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:32.016094 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:32.016443 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:32.016447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:32.016818 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:32.016975 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:32.016983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.017952 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.017957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:32.018969 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:32.018980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.019738 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.019744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:32.019902 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:32.020499 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:32.020516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:32.023771 (+ 3255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 20:10:32.024242 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:32.015854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:53523 (local address 127.4.169.190:34607)
0620 20:10:32.016029 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:32.016035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:32.016054 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:10:32.016248 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:32.016253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:32.016320 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:32.016420 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:32.017108 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.017807 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.019102 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.019568 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.019709 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:32.023969 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:53523: BlockingRecv error: recv got EOF from 127.4.169.131:53523 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 20:10:32 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:32 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:32.032791 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:32.024782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:42553)
0620 20:10:32.024937 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:32.024954 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:32.025061 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:32.025409 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:32.025414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:32.025710 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:32.025845 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:32.025853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.026834 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.026840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:32.027794 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:32.027804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.028547 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.028554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:32.028735 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:32.029268 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:32.029283 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:32.032500 (+ 3217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:10:32.032914 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:32.024939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:42553 (local address 127.4.169.190:34607)
0620 20:10:32.025173 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:32.025176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:32.025188 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:10:32.025221 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:32.025224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:32.025269 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:32.025381 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:32.025976 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.026711 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.027963 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.028390 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.028518 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:32.032654 (+ 4136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:42553: BlockingRecv error: recv got EOF from 127.4.169.131:42553 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260620 20:10:32.033245 14014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 20:10:32 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:32 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:32.467692 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:32.458202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:57869)
0620 20:10:32.458711 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:32.458730 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:32.458895 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:32.459236 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:32.459240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:32.459677 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:32.459825 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:32.459833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.460957 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.460966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:32.462005 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:32.462017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.462953 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.462958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:32.463165 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:32.463869 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:32.463893 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:32.467455 (+ 3562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
W20260620 20:10:32.467902 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:32.458321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:57869 (local address 127.4.169.190:34607)
0620 20:10:32.458562 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:32.458570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:32.458828 (+ 258us) server_negotiation.cc:407] Connection header received
0620 20:10:32.459022 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:32.459027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:32.459110 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:32.459211 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:32.459970 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.460731 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.462163 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.462762 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.462911 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:32.467665 (+ 4754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:57869: BlockingRecv error: recv got EOF from 127.4.169.129:57869 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 20:10:32 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:32 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:32.852721 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:32.843609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:33343)
0620 20:10:32.844140 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:32.844161 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:32.844323 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:32.844659 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:32.844663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:32.845212 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:32.845357 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:32.845366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.846333 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.846338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:32.847377 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:32.847388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.848139 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.848145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:32.848316 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:32.848912 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:32.848929 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:32.852493 (+ 3564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":310,"threads_started":1}
W20260620 20:10:32.852921 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:32.843700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:33343 (local address 127.4.169.190:34607)
0620 20:10:32.843947 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:32.843954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:32.844262 (+ 308us) server_negotiation.cc:407] Connection header received
0620 20:10:32.844460 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:32.844465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:32.844530 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:32.844626 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:32.845492 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.846200 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.847543 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:32.847965 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:32.848107 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:32.852672 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:33343: BlockingRecv error: recv got EOF from 127.4.169.130:33343 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 20:10:33 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:33 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:33.043046 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:33.033951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:53801)
0620 20:10:33.034527 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:33.034542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:33.034694 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:33.035060 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:33.035065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:33.035549 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:33.035691 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:33.035707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.036722 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.036727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:33.037771 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:33.037783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.038636 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.038643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:33.038827 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:33.039325 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:33.039340 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:33.042708 (+ 3368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":329,"threads_started":1}
W20260620 20:10:33.043130 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:33.034159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:53801 (local address 127.4.169.190:34607)
0620 20:10:33.034437 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:33.034444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:33.034617 (+ 173us) server_negotiation.cc:407] Connection header received
0620 20:10:33.034830 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:33.034835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:33.034917 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:33.035020 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:33.035830 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.036582 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.037906 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.038455 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.038600 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:33.042894 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:53801: BlockingRecv error: recv got EOF from 127.4.169.131:53801 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Jun 20 20:10:33 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:33 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:33.478209 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:33.468781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:42819)
0620 20:10:33.469302 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:33.469322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:33.469488 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:33.469841 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:33.469845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:33.470248 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:33.470445 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:33.470454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.471484 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.471492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:33.472417 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:33.472429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.473151 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.473157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:33.473342 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:33.473917 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:33.473935 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:33.477960 (+ 4025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":294,"threads_started":1}
W20260620 20:10:33.478451 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:33.468861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:42819 (local address 127.4.169.190:34607)
0620 20:10:33.469117 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:33.469125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:33.469411 (+ 286us) server_negotiation.cc:407] Connection header received
0620 20:10:33.469616 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:33.469622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:33.469696 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:33.469810 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:33.470601 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.471350 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.472558 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.472996 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.473132 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:33.478151 (+ 5019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:42819: BlockingRecv error: recv got EOF from 127.4.169.129:42819 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 20:10:33 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:33 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:33.862622 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:33.853637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:55505)
0620 20:10:33.854127 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:33.854166 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:33.854277 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:33.854584 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:33.854587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:33.855022 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:33.855190 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:33.855198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.856202 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.856206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:33.857219 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:33.857231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.857900 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.857904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:33.858019 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:33.858642 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:33.858665 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:33.862319 (+ 3654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":292,"threads_started":1}
W20260620 20:10:33.862700 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:33.853688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:55505 (local address 127.4.169.190:34607)
0620 20:10:33.853938 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:33.853946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:33.854222 (+ 276us) server_negotiation.cc:407] Connection header received
0620 20:10:33.854377 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:33.854380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:33.854464 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:33.854548 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:33.855324 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.856038 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.857337 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:33.857751 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:33.857889 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:33.862539 (+ 4650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:55505: BlockingRecv error: recv got EOF from 127.4.169.130:55505 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 20:10:34 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:34 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:34.053582 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:34.044039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:32901)
0620 20:10:34.044582 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:34.044598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:34.044763 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:34.045108 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:34.045112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:34.045565 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:34.045739 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:34.045748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.046927 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.046933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:34.047998 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:34.048010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.048709 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.048715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:34.048893 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:34.049530 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:34.049548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:34.053238 (+ 3690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":303,"threads_started":1}
W20260620 20:10:34.053706 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:34.044229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:32901 (local address 127.4.169.190:34607)
0620 20:10:34.044443 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:34.044449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:34.044678 (+ 229us) server_negotiation.cc:407] Connection header received
0620 20:10:34.044884 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:34.044890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:34.044977 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:34.045079 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:34.045878 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.046787 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.048138 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.048554 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.048693 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:34.053423 (+ 4730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:32901: BlockingRecv error: recv got EOF from 127.4.169.131:32901 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 20:10:34 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:34 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:34.487775 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:34.479261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:35387)
0620 20:10:34.479705 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:34.479726 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:34.479837 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:34.480108 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:34.480112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:34.480435 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:34.480592 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:34.480600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.481562 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.481565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:34.482560 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:34.482571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.483241 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.483245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:34.483367 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:34.483987 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:34.484005 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:34.487527 (+ 3522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
W20260620 20:10:34.487991 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:34.479361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:35387 (local address 127.4.169.190:34607)
0620 20:10:34.479590 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:34.479596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:34.479801 (+ 205us) server_negotiation.cc:407] Connection header received
0620 20:10:34.479918 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:34.479921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:34.479982 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:34.480065 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:34.480734 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.481452 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.482687 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.483122 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.483218 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:34.487730 (+ 4512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:35387: BlockingRecv error: recv got EOF from 127.4.169.129:35387 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 20:10:34 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:34 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:34.875437 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:34.863560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:58563)
0620 20:10:34.864011 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:34.864032 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:34.864172 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:34.864493 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:34.864497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:34.864878 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:34.865031 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:34.865039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.866066 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.866071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:34.867072 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:34.867083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.867776 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.867781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:34.867957 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:34.871549 (+ 3592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:34.871567 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:34.875167 (+ 3600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":237,"threads_started":1}
W20260620 20:10:34.875622 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:34.863641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:58563 (local address 127.4.169.190:34607)
0620 20:10:34.863858 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:34.863865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:34.864106 (+ 241us) server_negotiation.cc:407] Connection header received
0620 20:10:34.864301 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:34.864306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:34.864372 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:34.864468 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:34.865172 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.865925 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.867207 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:34.867630 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:34.867760 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:34.875361 (+ 7601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:58563: BlockingRecv error: recv got EOF from 127.4.169.130:58563 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 20:10:35 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:35 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:35.062775 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:35.054499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:47885)
0620 20:10:35.055069 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:35.055088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:35.055218 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:35.055547 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:35.055551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:35.055893 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:35.056073 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:35.056081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.057107 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.057111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:35.058016 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:35.058026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.058647 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.058651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:35.058777 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:35.059327 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:35.059343 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:35.062530 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":325,"threads_started":1}
W20260620 20:10:35.062950 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:35.054587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:47885 (local address 127.4.169.190:34607)
0620 20:10:35.054745 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:35.054749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:35.055172 (+ 423us) server_negotiation.cc:407] Connection header received
0620 20:10:35.055301 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:35.055304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:35.055376 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:35.055457 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:35.056246 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.056992 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.058127 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.058529 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.058620 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:35.062725 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:47885: BlockingRecv error: recv got EOF from 127.4.169.131:47885 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 20:10:35 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:35 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:35.497395 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:35.488769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:48831)
0620 20:10:35.489225 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:35.489245 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:35.489351 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:35.489632 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:35.489635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:35.489993 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:35.490150 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:35.490159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.491191 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.491195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:35.492185 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:35.492196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.492821 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.492825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:35.492956 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:35.493509 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:35.493525 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:35.497138 (+ 3613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":264,"threads_started":1}
W20260620 20:10:35.497601 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:35.488839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:48831 (local address 127.4.169.190:34607)
0620 20:10:35.489127 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:35.489133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:35.489340 (+ 207us) server_negotiation.cc:407] Connection header received
0620 20:10:35.489450 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:35.489454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:35.489516 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:35.489594 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:35.490283 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.491046 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.492309 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.492706 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.492795 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:35.497349 (+ 4554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:48831: BlockingRecv error: recv got EOF from 127.4.169.129:48831 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 20:10:35 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:35 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:35.884683 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:35.876401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:49103)
0620 20:10:35.876825 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:35.876846 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:35.876963 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:35.877216 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:35.877219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:35.877572 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:35.877741 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:35.877750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.878681 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.878685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:35.879667 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:35.879677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.880305 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.880308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:35.880436 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:35.880993 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:35.881009 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:35.884293 (+ 3284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":241,"threads_started":1}
W20260620 20:10:35.884738 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:35.876440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:49103 (local address 127.4.169.190:34607)
0620 20:10:35.876651 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:35.876661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:35.876933 (+ 272us) server_negotiation.cc:407] Connection header received
0620 20:10:35.877042 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:35.877046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:35.877104 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:35.877178 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:35.877856 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.878564 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.879783 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:35.880187 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:35.880284 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:35.884520 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:49103: BlockingRecv error: recv got EOF from 127.4.169.130:49103 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 20:10:36 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:36 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:36.072481 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:36.063721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:56117)
0620 20:10:36.064217 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:36.064233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:36.064382 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:36.064748 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:36.064753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:36.065133 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:36.065286 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:36.065295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.066262 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.066269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:36.067343 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:36.067354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.068049 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.068055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:36.068232 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:36.068775 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:36.068792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:36.072121 (+ 3329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":262,"threads_started":1}
W20260620 20:10:36.072618 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:36.063791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:56117 (local address 127.4.169.190:34607)
0620 20:10:36.064007 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:36.064016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:36.064315 (+ 299us) server_negotiation.cc:407] Connection header received
0620 20:10:36.064499 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:36.064504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:36.064596 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:36.064694 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:36.065422 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.066122 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.067496 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.067899 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.068025 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:36.072332 (+ 4307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:56117: BlockingRecv error: recv got EOF from 127.4.169.131:56117 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 20:10:36 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:36 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:36.507061 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:36.498268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:54997)
0620 20:10:36.498758 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:36.498779 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:36.498885 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:36.499164 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:36.499168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:36.499592 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:36.499735 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:36.499743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.500644 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.500647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:36.501571 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:36.501582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.502431 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.502437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:36.502613 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:36.503240 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:36.503266 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:36.506729 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":276,"threads_started":1}
W20260620 20:10:36.507133 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:36.498433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:54997 (local address 127.4.169.190:34607)
0620 20:10:36.498614 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:36.498622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:36.498857 (+ 235us) server_negotiation.cc:407] Connection header received
0620 20:10:36.498981 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:36.498984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:36.499040 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:36.499117 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:36.499855 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.500532 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.501709 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.502284 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.502423 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:36.506898 (+ 4475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:54997: BlockingRecv error: recv got EOF from 127.4.169.129:54997 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 20:10:36 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:36 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:36.893971 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:36.885605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:40233)
0620 20:10:36.886028 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:36.886049 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:36.886156 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:36.886445 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:36.886448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:36.886812 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:36.886961 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:36.886968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.887929 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.887933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:36.888848 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:36.888859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.889476 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.889479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:36.889595 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:36.890213 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:36.890230 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:36.893669 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":235,"threads_started":1}
W20260620 20:10:36.894107 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:36.885797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:40233 (local address 127.4.169.190:34607)
0620 20:10:36.886021 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:36.886026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:36.886120 (+ 94us) server_negotiation.cc:407] Connection header received
0620 20:10:36.886251 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:36.886254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:36.886322 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:36.886425 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:36.887082 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.887825 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.888959 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:36.889368 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:36.889455 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:36.893845 (+ 4390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:40233: BlockingRecv error: recv got EOF from 127.4.169.130:40233 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 20:10:37 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:37 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:37.082464 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:37.073472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:42579)
0620 20:10:37.074018 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:37.074036 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:37.074151 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:37.074478 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:37.074482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:37.074884 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:37.075038 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:37.075046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.076015 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.076020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:37.076947 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:37.076958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.077700 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.077704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:37.077835 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:37.078436 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:37.078454 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:37.082091 (+ 3637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":310,"threads_started":1}
W20260620 20:10:37.082584 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:37.073562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:42579 (local address 127.4.169.190:34607)
0620 20:10:37.073731 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:37.073746 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:37.074107 (+ 361us) server_negotiation.cc:407] Connection header received
0620 20:10:37.074233 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:37.074238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:37.074311 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:37.074441 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:37.075158 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.075876 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.077103 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.077543 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.077648 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:37.082302 (+ 4654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:42579: BlockingRecv error: recv got EOF from 127.4.169.131:42579 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 20:10:37 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:37 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:37.517542 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:37.508091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:42023)
0620 20:10:37.508697 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:37.508716 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:37.508895 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:37.509241 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:37.509246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:37.509658 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:37.509798 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:37.509806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.510863 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.510869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:37.511827 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:37.511839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.512855 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.512861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:37.513066 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:37.513630 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:37.513647 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:37.517309 (+ 3662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":347,"threads_started":1}
W20260620 20:10:37.517784 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:37.508282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:42023 (local address 127.4.169.190:34607)
0620 20:10:37.508551 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:37.508559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:37.508810 (+ 251us) server_negotiation.cc:407] Connection header received
0620 20:10:37.509024 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:37.509029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:37.509100 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:37.509206 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:37.509940 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.510720 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.511972 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.512669 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.512840 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:37.517538 (+ 4698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:42023: BlockingRecv error: recv got EOF from 127.4.169.129:42023 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 20:10:37 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:37 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:37.904085 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:37.895035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:51381)
0620 20:10:37.895557 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:37.895579 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:37.895690 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:37.896051 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:37.896055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:37.896469 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:37.896646 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:37.896655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.897631 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.897634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:37.898631 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:37.898643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.899643 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.899648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:37.899833 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:37.900519 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:37.900535 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:37.903860 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":293,"threads_started":1}
W20260620 20:10:37.904366 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:37.895138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:51381 (local address 127.4.169.190:34607)
0620 20:10:37.895322 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:37.895327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:37.895650 (+ 323us) server_negotiation.cc:407] Connection header received
0620 20:10:37.895811 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:37.895816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:37.895916 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:37.896016 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:37.896785 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.897511 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.898774 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:37.899494 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:37.899623 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:37.904057 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:51381: BlockingRecv error: recv got EOF from 127.4.169.130:51381 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 20:10:38 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:38 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:38.092486 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:38.083525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:57159)
0620 20:10:38.084099 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:38.084120 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:38.084247 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:38.084653 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:38.084656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:38.085035 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:38.085242 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:38.085250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.086250 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.086254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:38.087225 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:38.087237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.087893 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.087899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:38.088026 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:38.088627 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:38.088643 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:38.092235 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":274,"threads_started":1}
W20260620 20:10:38.092767 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:38.083673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:57159 (local address 127.4.169.190:34607)
0620 20:10:38.083898 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:38.083904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:38.084223 (+ 319us) server_negotiation.cc:407] Connection header received
0620 20:10:38.084412 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:38.084418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:38.084512 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:38.084605 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:38.085367 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.086119 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.087347 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.087731 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.087820 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:38.092458 (+ 4638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:57159: BlockingRecv error: recv got EOF from 127.4.169.131:57159 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 20:10:38 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:38 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:38.527712 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:38.518610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:42175)
0620 20:10:38.519149 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:38.519170 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:38.519283 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:38.519567 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:38.519570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:38.519977 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:38.520165 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:38.520173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.521185 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.521189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:38.522149 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:38.522160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.522866 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.522870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:38.522979 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:38.523511 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:38.523528 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:38.527453 (+ 3925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":244,"threads_started":1}
W20260620 20:10:38.527804 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:38.518670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:42175 (local address 127.4.169.190:34607)
0620 20:10:38.518899 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:38.518909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:38.519256 (+ 347us) server_negotiation.cc:407] Connection header received
0620 20:10:38.519383 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:38.519386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:38.519449 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:38.519533 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:38.520330 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.521071 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.522282 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.522743 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.522844 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:38.527642 (+ 4798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:42175: BlockingRecv error: recv got EOF from 127.4.169.129:42175 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 20:10:38 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:38 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:38.914837 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:38.905267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:55015)
0620 20:10:38.905897 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:38.905918 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:38.906032 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:38.906328 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:38.906331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:38.906769 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:38.906944 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:38.906953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.907975 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.907979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:38.908927 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:38.908938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.909611 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.909615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:38.909724 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:38.910473 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:38.910493 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:38.914587 (+ 4094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":286,"threads_started":1}
W20260620 20:10:38.915130 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:38.905348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:55015 (local address 127.4.169.190:34607)
0620 20:10:38.905551 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:38.905556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:38.905990 (+ 434us) server_negotiation.cc:407] Connection header received
0620 20:10:38.906134 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:38.906138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:38.906200 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:38.906287 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:38.907125 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.907866 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.909055 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:38.909478 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:38.909570 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:38.914799 (+ 5229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:55015: BlockingRecv error: recv got EOF from 127.4.169.130:55015 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 20:10:39 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:39 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:39.102696 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:39.093682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:59137)
0620 20:10:39.094321 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:39.094338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:39.094481 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:39.094797 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:39.094800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:39.095288 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:39.095460 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:39.095469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.096527 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.096531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:39.097466 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:39.097477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.098217 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.098221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:39.098444 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:39.099111 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:39.099128 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:39.102447 (+ 3319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":309,"threads_started":1}
W20260620 20:10:39.102885 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:39.093753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:59137 (local address 127.4.169.190:34607)
0620 20:10:39.093980 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:39.093990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:39.094421 (+ 431us) server_negotiation.cc:407] Connection header received
0620 20:10:39.094584 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:39.094588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:39.094651 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:39.094753 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:39.095593 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.096398 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.097591 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.098072 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.098196 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:39.102629 (+ 4433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:59137: BlockingRecv error: recv got EOF from 127.4.169.131:59137 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 20:10:39 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:39 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:39.537792 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:39.528698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:39667)
0620 20:10:39.529257 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:39.529285 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:39.529427 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:39.529956 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:39.529961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:39.530572 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:39.530788 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:39.530801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.531790 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.531793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:39.532768 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:39.532780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.533456 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.533460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:39.533574 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:39.534223 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:39.534240 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:39.537546 (+ 3306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":260,"threads_started":1}
W20260620 20:10:39.537985 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:39.528983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:39667 (local address 127.4.169.190:34607)
0620 20:10:39.529179 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:39.529183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:39.529384 (+ 201us) server_negotiation.cc:407] Connection header received
0620 20:10:39.529706 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:39.529711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:39.529812 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:39.529891 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:39.530929 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.531664 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.532894 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.533301 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.533394 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:39.537744 (+ 4350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:39667: BlockingRecv error: recv got EOF from 127.4.169.129:39667 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 20:10:39 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:39 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:39.930190 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:39.915852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:57115)
0620 20:10:39.916412 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:39.916430 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:39.916588 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:39.916937 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:39.916941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:39.917355 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:39.917517 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:39.917525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.918748 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.918754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:39.919682 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:39.919693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.920468 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.920474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:39.920637 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:39.926011 (+ 5374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:39.926032 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:39.929933 (+ 3901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":270,"threads_started":1}
W20260620 20:10:39.930452 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:39.916033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:57115 (local address 127.4.169.190:34607)
0620 20:10:39.916290 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:39.916297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:39.916517 (+ 220us) server_negotiation.cc:407] Connection header received
0620 20:10:39.916722 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:39.916727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:39.916797 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:39.916908 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:39.917658 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.918604 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.919831 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:39.920286 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:39.920428 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:39.930190 (+ 9762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:57115: BlockingRecv error: recv got EOF from 127.4.169.130:57115 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 20:10:40 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:40 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:40.112951 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:40.103771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:53985)
0620 20:10:40.104342 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:40.104359 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:40.104482 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:40.104832 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:40.104837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:40.105341 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:40.105565 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:40.105580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.106938 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.106942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:40.107875 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:40.107885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.108563 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.108567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:40.108682 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:40.109424 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:40.109441 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:40.112718 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":309,"threads_started":1}
W20260620 20:10:40.113165 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:40.103931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:53985 (local address 127.4.169.190:34607)
0620 20:10:40.104124 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:40.104128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:40.104452 (+ 324us) server_negotiation.cc:407] Connection header received
0620 20:10:40.104584 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:40.104588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:40.104654 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:40.104751 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:40.105731 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.106804 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.107997 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.108416 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.108507 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:40.112899 (+ 4392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:53985: BlockingRecv error: recv got EOF from 127.4.169.131:53985 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 20:10:40 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:40 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:40.547950 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:40.538853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:55507)
0620 20:10:40.539357 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:40.539377 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:40.539531 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:40.539898 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:40.539902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:40.540367 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:40.540599 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:40.540612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.541734 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.541742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:40.542756 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:40.542767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.543442 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.543448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:40.543617 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:40.544236 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:40.544253 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:40.547623 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":288,"threads_started":1}
W20260620 20:10:40.548043 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:40.538909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:55507 (local address 127.4.169.190:34607)
0620 20:10:40.539152 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:40.539160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:40.539491 (+ 331us) server_negotiation.cc:407] Connection header received
0620 20:10:40.539678 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:40.539682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:40.539745 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:40.539863 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:40.540741 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.541601 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.542886 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.543295 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.543429 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:40.547807 (+ 4378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:55507: BlockingRecv error: recv got EOF from 127.4.169.129:55507 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 20:10:40 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:40 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:40.940148 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:40.931156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:50495)
0620 20:10:40.931672 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:40.931693 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:40.931808 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:40.932087 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:40.932090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:40.932472 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:40.932608 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:40.932616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.933833 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.933838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:40.934938 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:40.934950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.935690 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.935695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:40.935818 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:40.936372 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:40.936389 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:40.939795 (+ 3406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":276,"threads_started":1}
W20260620 20:10:40.940284 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:40.931288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:50495 (local address 127.4.169.190:34607)
0620 20:10:40.931457 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:40.931461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:40.931773 (+ 312us) server_negotiation.cc:407] Connection header received
0620 20:10:40.931901 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:40.931905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:40.931963 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:40.932052 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:40.932729 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.933660 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.935098 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:40.935553 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:40.935647 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:40.940009 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:50495: BlockingRecv error: recv got EOF from 127.4.169.130:50495 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 20:10:41 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:41 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:41.123191 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:41.113944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:37061)
0620 20:10:41.114496 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:41.114513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:41.114671 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:41.115013 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:41.115020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:41.115432 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:41.115571 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:41.115580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.116558 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.116566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:41.117510 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:41.117521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.118240 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.118246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:41.118450 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:41.119001 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:41.119017 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:41.122865 (+ 3848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":288,"threads_started":1}
W20260620 20:10:41.123323 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:41.114013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:37061 (local address 127.4.169.190:34607)
0620 20:10:41.114278 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:41.114285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:41.114605 (+ 320us) server_negotiation.cc:407] Connection header received
0620 20:10:41.114800 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:41.114806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:41.114872 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:41.114975 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:41.115722 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.116424 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.117649 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.118080 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.118217 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:41.123044 (+ 4827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:37061: BlockingRecv error: recv got EOF from 127.4.169.131:37061 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 20:10:41 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:41 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:41.558030 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:41.548942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:59977)
0620 20:10:41.549408 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:41.549428 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:41.549544 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:41.549835 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:41.549839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:41.550186 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:41.550330 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:41.550338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.551402 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.551406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:41.552440 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:41.552454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.553190 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.553194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:41.553310 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:41.553911 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:41.553929 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:41.557791 (+ 3862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":262,"threads_started":1}
W20260620 20:10:41.558259 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:41.548999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:59977 (local address 127.4.169.190:34607)
0620 20:10:41.549246 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:41.549253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:41.549510 (+ 257us) server_negotiation.cc:407] Connection header received
0620 20:10:41.549645 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:41.549648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:41.549709 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:41.549791 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:41.550520 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.551276 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.552594 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.553048 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.553151 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:41.557975 (+ 4824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:59977: BlockingRecv error: recv got EOF from 127.4.169.129:59977 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 20:10:41 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:41 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:41.949882 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:41.941229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:54723)
0620 20:10:41.941718 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:41.941739 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:41.941891 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:41.942211 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:41.942216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:41.942619 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:41.942759 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:41.942767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.943751 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.943760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:41.944726 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:41.944738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.945410 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.945416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:41.945583 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:41.946233 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:41.946251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:41.949515 (+ 3264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":263,"threads_started":1}
W20260620 20:10:41.949950 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:41.941307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:54723 (local address 127.4.169.190:34607)
0620 20:10:41.941517 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:41.941526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:41.941823 (+ 297us) server_negotiation.cc:407] Connection header received
0620 20:10:41.942011 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:41.942016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:41.942083 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:41.942184 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:41.942902 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.943614 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.944864 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:41.945262 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:41.945395 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:41.949701 (+ 4306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:54723: BlockingRecv error: recv got EOF from 127.4.169.130:54723 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 20:10:42 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:42 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:42.133108 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:42.124210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:50193)
0620 20:10:42.124788 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:42.124804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:42.124964 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:42.125323 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:42.125328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:42.125743 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:42.125896 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:42.125905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.127033 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.127041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:42.128018 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:42.128028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.128778 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.128784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:42.128956 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:42.129589 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:42.129606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:42.132881 (+ 3275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":321,"threads_started":1}
W20260620 20:10:42.133347 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:42.124447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:50193 (local address 127.4.169.190:34607)
0620 20:10:42.124669 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:42.124676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:42.124892 (+ 216us) server_negotiation.cc:407] Connection header received
0620 20:10:42.125091 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:42.125098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:42.125163 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:42.125291 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:42.126063 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.126881 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.128161 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.128622 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.128766 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:42.133059 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:50193: BlockingRecv error: recv got EOF from 127.4.169.131:50193 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 20:10:42 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:42 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:42.567757 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:42.558955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:38115)
0620 20:10:42.559521 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:42.559548 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:42.559676 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:42.560039 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:42.560043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:42.560483 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:42.560657 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:42.560666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.561635 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.561642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:42.562571 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:42.562583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.563300 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.563306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:42.563495 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:42.564130 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:42.564148 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:42.567439 (+ 3291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":304,"threads_started":1}
W20260620 20:10:42.567831 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:42.559088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:38115 (local address 127.4.169.190:34607)
0620 20:10:42.559334 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:42.559341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:42.559644 (+ 303us) server_negotiation.cc:407] Connection header received
0620 20:10:42.559824 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:42.559827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:42.559889 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:42.560008 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:42.560805 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.561498 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.562729 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.563151 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.563289 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:42.567603 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:38115: BlockingRecv error: recv got EOF from 127.4.169.129:38115 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 20:10:42 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:42 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:42.960130 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:42.950918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:39609)
0620 20:10:42.951426 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:42.951447 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:42.951600 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:42.951932 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:42.951937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:42.952377 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:42.952539 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:42.952548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.953582 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.953589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:42.954659 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:42.954671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.955484 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.955489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:42.955655 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:42.956336 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:42.956354 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:42.959795 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":298,"threads_started":1}
W20260620 20:10:42.960222 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:42.951045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:39609 (local address 127.4.169.190:34607)
0620 20:10:42.951218 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:42.951224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:42.951532 (+ 308us) server_negotiation.cc:407] Connection header received
0620 20:10:42.951739 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:42.951743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:42.951804 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:42.951905 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:42.952678 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.953451 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.954798 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:42.955340 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:42.955469 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:42.959970 (+ 4501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:39609: BlockingRecv error: recv got EOF from 127.4.169.130:39609 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 20:10:43 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:43 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:43.143051 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:43.133984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:58929)
0620 20:10:43.134615 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:43.134631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:43.134786 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:43.135106 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:43.135111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:43.135597 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:43.135778 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:43.135786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.136769 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.136777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:43.137881 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:43.137893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.138622 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.138628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:43.138800 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:43.139415 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:43.139433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:43.142832 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":435,"threads_started":1}
W20260620 20:10:43.143250 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:43.134045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:58929 (local address 127.4.169.190:34607)
0620 20:10:43.134204 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:43.134209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:43.134719 (+ 510us) server_negotiation.cc:407] Connection header received
0620 20:10:43.134908 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:43.134919 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:43.134978 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:43.135079 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:43.135911 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.136627 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.138019 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.138478 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.138607 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:43.143000 (+ 4393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:58929: BlockingRecv error: recv got EOF from 127.4.169.131:58929 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 20:10:43 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:43 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:43.577993 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:43.568777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:45101)
0620 20:10:43.569344 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:43.569364 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:43.569474 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:43.569786 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:43.569789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:43.570169 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:43.570355 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:43.570363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.571426 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.571430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:43.572411 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:43.572424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.573137 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.573141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:43.573289 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:43.573908 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:43.573926 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:43.577746 (+ 3820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":323,"threads_started":1}
W20260620 20:10:43.578164 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:43.568830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:45101 (local address 127.4.169.190:34607)
0620 20:10:43.569008 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:43.569012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:43.569436 (+ 424us) server_negotiation.cc:407] Connection header received
0620 20:10:43.569581 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:43.569584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:43.569645 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:43.569737 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:43.570536 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.571280 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.572549 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.572998 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.573089 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:43.577914 (+ 4825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:45101: BlockingRecv error: recv got EOF from 127.4.169.129:45101 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 20:10:43 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:43 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:43.969851 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:43.961247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:40247)
0620 20:10:43.961723 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:43.961746 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:43.961894 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:43.962216 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:43.962238 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:43.962649 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:43.962811 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:43.962819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.963812 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.963819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:43.964734 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:43.964745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.965413 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.965418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:43.965606 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:43.966198 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:43.966214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:43.969512 (+ 3298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":261,"threads_started":1}
W20260620 20:10:43.969947 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:43.961321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:40247 (local address 127.4.169.190:34607)
0620 20:10:43.961550 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:43.961558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:43.961831 (+ 273us) server_negotiation.cc:407] Connection header received
0620 20:10:43.962014 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:43.962019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:43.962088 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:43.962186 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:43.962948 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.963670 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.964869 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:43.965276 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:43.965404 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:43.969687 (+ 4283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:40247: BlockingRecv error: recv got EOF from 127.4.169.130:40247 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 20:10:44 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:44 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:44.153173 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:44.143986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:53361)
0620 20:10:44.144524 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:44.144540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:44.144648 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:44.144984 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:44.144988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:44.145399 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:44.145570 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:44.145578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.146662 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.146666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:44.147682 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:44.147695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.148370 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.148374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:44.148504 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:44.149058 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:44.149075 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:44.152841 (+ 3766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":296,"threads_started":1}
W20260620 20:10:44.153316 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:44.144028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:53361 (local address 127.4.169.190:34607)
0620 20:10:44.144193 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:44.144198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:44.144607 (+ 409us) server_negotiation.cc:407] Connection header received
0620 20:10:44.144740 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:44.144743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:44.144819 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:44.144897 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:44.145707 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.146543 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.147799 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.148196 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.148296 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:44.153045 (+ 4749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:53361: BlockingRecv error: recv got EOF from 127.4.169.131:53361 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 20:10:44 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:44 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:44.587746 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:44.578982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:55387)
0620 20:10:44.579508 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:44.579533 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:44.579672 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:44.580047 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:44.580052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:44.580468 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:44.580607 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:44.580616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.581535 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.581538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:44.582794 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:44.582808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.583456 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.583460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:44.583595 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:44.584162 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:44.584180 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:44.587517 (+ 3337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":290,"threads_started":1}
W20260620 20:10:44.587879 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:44.579085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:55387 (local address 127.4.169.190:34607)
0620 20:10:44.579352 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:44.579359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:44.579644 (+ 285us) server_negotiation.cc:407] Connection header received
0620 20:10:44.579811 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:44.579818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:44.579901 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:44.579976 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:44.580731 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.581425 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.582932 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.583326 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.583416 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:44.587689 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:55387: BlockingRecv error: recv got EOF from 127.4.169.129:55387 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 20:10:44 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:44 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.130@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:44.983939 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:44.970929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.130:39313)
0620 20:10:44.971429 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:44.971460 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:44.971568 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:44.971852 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:44.971855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:44.972224 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:44.972366 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:44.972373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.973315 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.973319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:44.974215 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:44.974227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.974892 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.974896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:44.975015 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:44.979972 (+ 4957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:44.979989 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:44.983696 (+ 3707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":289,"threads_started":1}
W20260620 20:10:44.984153 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:44.971049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:39313 (local address 127.4.169.190:34607)
0620 20:10:44.971247 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:44.971252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:44.971531 (+ 279us) server_negotiation.cc:407] Connection header received
0620 20:10:44.971658 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:44.971661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:44.971732 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:44.971811 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:44.972479 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.973199 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.974337 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:44.974779 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:44.974868 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:44.983900 (+ 9032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.130:39313: BlockingRecv error: recv got EOF from 127.4.169.130:39313 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 20:10:45 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:45 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.131@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:45.163064 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:45.154214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.131:46125)
0620 20:10:45.154692 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:45.154708 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:45.154858 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:45.155241 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:45.155246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:45.155657 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:45.155809 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:45.155818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.156817 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.156823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:45.157733 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:45.157743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.158457 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.158462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:45.158681 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:45.159293 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:45.159311 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:45.162742 (+ 3431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":256,"threads_started":1}
W20260620 20:10:45.163136 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:45.154290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:46125 (local address 127.4.169.190:34607)
0620 20:10:45.154525 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:45.154532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:45.154788 (+ 256us) server_negotiation.cc:407] Connection header received
0620 20:10:45.155021 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:45.155025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:45.155084 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:45.155213 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:45.155947 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.156675 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.157871 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.158289 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.158446 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:45.162916 (+ 4470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.131:46125: BlockingRecv error: recv got EOF from 127.4.169.131:46125 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 20:10:45 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:10:45 dist-test-slave-4bwq krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.169.129@KRBTEST.COM for oryx/127.4.169.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:10:45.597961 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:45.588938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34607 (local address 127.4.169.129:51101)
0620 20:10:45.589423 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:45.589443 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:45.589592 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:45.589946 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:45.589950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:45.590353 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:45.590551 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:45.590559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.591556 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.591562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:45.592685 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:45.592697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.593397 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.593402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:45.593573 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:45.594171 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:45.594195 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:45.597611 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.169.190:34607: Server oryx/127.4.169.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
W20260620 20:10:45.598089 14019 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:45.589029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:51101 (local address 127.4.169.190:34607)
0620 20:10:45.589313 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:45.589319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:45.589526 (+ 207us) server_negotiation.cc:407] Connection header received
0620 20:10:45.589728 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:45.589732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:45.589818 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:45.589922 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:45.590713 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.591409 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.592821 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.593256 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.593385 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:45.597797 (+ 4412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.169.129:51101: BlockingRecv error: recv got EOF from 127.4.169.129:51101 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
I20260620 20:10:45.849009 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13601
I20260620 20:10:45.862154 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:45.850114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.4.169.129:41259)
0620 20:10:45.850827 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:45.850834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:45.850863 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:10:45.850921 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:45.850928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:45.851152 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:45.851311 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:45.852688 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.854569 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.855858 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.856530 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.856633 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:45.858993 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:45.859002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:45.859015 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:45.859068 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:45.861620 (+ 2552us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.169.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 20:10:45.861888 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47764: Request ticket server kudu/127.4.169.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":371,"threads_started":1}
W20260620 20:10:45.862269 14062 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47764: Request ticket server kudu/127.4.169.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 20:10:45.862290 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13601: Not authorized: Client connection negotiation failed: client connection to 127.4.169.129:41259: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.169.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 20:10:45.862568 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 13601
I20260620 20:10:45.879135 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13752
I20260620 20:10:45.891261 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:45.880236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.4.169.130:46033)
0620 20:10:45.880796 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:45.880803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:45.880828 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:10:45.880890 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:45.880897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:45.881124 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:45.881289 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:45.883075 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.884026 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.885186 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.885736 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.885836 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:45.888268 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:45.888276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:45.888290 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:45.888326 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:45.890753 (+ 2427us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.169.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 20:10:45.891023 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57112: Request ticket server kudu/127.4.169.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":254,"threads_started":1}
W20260620 20:10:45.891388 14063 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57112: Request ticket server kudu/127.4.169.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 20:10:45.891477 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13752: Not authorized: Client connection negotiation failed: client connection to 127.4.169.130:46033: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.169.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 20:10:45.891685 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 13752
I20260620 20:10:45.907950 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13885
I20260620 20:10:45.921329 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:45.909021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.4.169.131:43587)
0620 20:10:45.910822 (+ 1801us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:45.910829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:45.910856 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:10:45.910917 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:45.910923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:45.911148 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:45.911361 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:45.912722 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.913689 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.915041 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:45.915605 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:45.915760 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:45.918248 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:45.918257 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:45.918270 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:45.918317 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:45.920609 (+ 2292us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.169.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 20:10:45.920968 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50538: Request ticket server kudu/127.4.169.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":1522,"threads_started":1}
W20260620 20:10:45.921327 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13885: Not authorized: Client connection negotiation failed: client connection to 127.4.169.131:43587: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.169.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 20:10:45.921438 14064 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50538: Request ticket server kudu/127.4.169.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 20:10:45.921470 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 13885
I20260620 20:10:45.933836 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 13532
I20260620 20:10:45.935055 13594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:46.046765 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 13532
2026-06-20T20:10:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20998 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T20:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:10:46Z Disabled control of system clock
I20260620 20:10:46.121733 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:39961
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:39961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:10:46.440007 14069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:46.440280 14069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:46.440358 14069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:46.447362 14069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:46.447468 14069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:46.447506 14069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:46.447553 14069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:46.447597 14069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:46.456550 14069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:39961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:39961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:46.457757 14069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:46.459410 14069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:46.471247 14077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:46.471326 14074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:46.471304 14075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:46.471882 14069 server_base.cc:1034] running on GCE node
I20260620 20:10:46.472623 14069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:46.473932 14069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:46.475152 14069 hybrid_clock.cc:648] HybridClock initialized: now 1781986246475115 us; error 33 us; skew 500 ppm
I20260620 20:10:46.475451 14069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:10:46.477494 14069 webserver.cc:460] Webserver started at http://127.4.169.190:42925/ using document root <none> and password file <none>
I20260620 20:10:46.478129 14069 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:46.478210 14069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:46.478561 14069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:46.481274 14069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "a75b26f7de7a4535beadd80aaf707547"
format_stamp: "Formatted at 2026-06-20 20:10:46 on dist-test-slave-4bwq"
server_key: "179a4fa3750cdc3460c624c2a609e3a8"
server_key_iv: "964a0f577e2793a783cd0c3619f37c5d"
server_key_version: "encryptionkey@0"
I20260620 20:10:46.481997 14069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "a75b26f7de7a4535beadd80aaf707547"
format_stamp: "Formatted at 2026-06-20 20:10:46 on dist-test-slave-4bwq"
server_key: "179a4fa3750cdc3460c624c2a609e3a8"
server_key_iv: "964a0f577e2793a783cd0c3619f37c5d"
server_key_version: "encryptionkey@0"
I20260620 20:10:46.486930 14069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:10:46.490063 14083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:46.491490 14069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:10:46.491650 14069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "a75b26f7de7a4535beadd80aaf707547"
format_stamp: "Formatted at 2026-06-20 20:10:46 on dist-test-slave-4bwq"
server_key: "179a4fa3750cdc3460c624c2a609e3a8"
server_key_iv: "964a0f577e2793a783cd0c3619f37c5d"
server_key_version: "encryptionkey@0"
I20260620 20:10:46.491828 14069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:46.503191 14069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:46.503981 14069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:46.504227 14069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:46.521207 14069 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:39961
I20260620 20:10:46.521301 14134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:39961 every 8 connection(s)
I20260620 20:10:46.522641 14069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:10:46.526583 14135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:46.531622 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 14069
I20260620 20:10:46.531886 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:10:46.532559 4774 external_mini_cluster.cc:1428] Setting key 3db065895f26f61e4aec0ee88c23c982
I20260620 20:10:46.538143 14135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547: Bootstrap starting.
I20260620 20:10:46.541989 14135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:46.543051 14135 log.cc:824] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:46.545991 14135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547: No bootstrap required, opened a new log
I20260620 20:10:46.547889 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:46.534663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.4.169.190:39961)
0620 20:10:46.535799 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:46.535817 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:46.535870 (+ 53us) server_negotiation.cc:407] Connection header received
0620 20:10:46.538086 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:46.538113 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:46.538733 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:46.539152 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:46.539686 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:46.541657 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:46.543171 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:46.544211 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:46.544426 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:46.545566 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:46.545574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:46.545587 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:46.546139 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:46.546237 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:46.546260 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:46.546392 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:46.546684 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 20:10:46.546974 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":364,"threads_started":1}
I20260620 20:10:46.553979 14135 raft_consensus.cc:348] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b26f7de7a4535beadd80aaf707547" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39961 } }
I20260620 20:10:46.554333 14135 raft_consensus.cc:374] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:46.554458 14135 raft_consensus.cc:729] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75b26f7de7a4535beadd80aaf707547, State: Initialized, Role: FOLLOWER
I20260620 20:10:46.555017 14135 consensus_queue.cc:260] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b26f7de7a4535beadd80aaf707547" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39961 } }
I20260620 20:10:46.555267 14135 raft_consensus.cc:388] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:10:46.555400 14135 raft_consensus.cc:482] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:10:46.555581 14135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:46.558208 14135 raft_consensus.cc:504] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b26f7de7a4535beadd80aaf707547" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39961 } }
I20260620 20:10:46.558727 14135 leader_election.cc:302] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a75b26f7de7a4535beadd80aaf707547; no voters:
I20260620 20:10:46.559320 14135 leader_election.cc:288] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:10:46.559451 14140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:46.560148 14140 raft_consensus.cc:686] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [term 1 LEADER]: Becoming Leader. State: Replica: a75b26f7de7a4535beadd80aaf707547, State: Running, Role: LEADER
I20260620 20:10:46.560705 14140 consensus_queue.cc:237] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b26f7de7a4535beadd80aaf707547" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39961 } }
I20260620 20:10:46.561247 14135 sys_catalog.cc:564] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:46.564623 14142 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a75b26f7de7a4535beadd80aaf707547. Latest consensus state: current_term: 1 leader_uuid: "a75b26f7de7a4535beadd80aaf707547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b26f7de7a4535beadd80aaf707547" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39961 } } }
I20260620 20:10:46.564656 14141 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a75b26f7de7a4535beadd80aaf707547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b26f7de7a4535beadd80aaf707547" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39961 } } }
I20260620 20:10:46.565042 14142 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:46.565097 14141 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:46.566202 14151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:46.571571 14151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:46.579905 14151 catalog_manager.cc:1347] Generated new cluster ID: 01bdf29074c94104adbc1cf2637f3698
I20260620 20:10:46.580027 14151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:46.592033 14151 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:10:46.593648 14151 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:46.603993 14151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a75b26f7de7a4535beadd80aaf707547: Generated new TSK 0
I20260620 20:10:46.604862 14151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:10:46.614934 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:39961
--builtin_ntp_servers=127.4.169.148:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 20:10:46.977952 14159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:46.978245 14159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:46.978328 14159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:46.985592 14159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:46.985704 14159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:46.985810 14159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:10:46.995309 14159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:46.996665 14159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:46.998320 14159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:47.011540 14164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:47.011487 14167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:47.011561 14165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:47.012341 14159 server_base.cc:1034] running on GCE node
I20260620 20:10:47.012997 14159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:47.013876 14159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:47.015097 14159 hybrid_clock.cc:648] HybridClock initialized: now 1781986247015045 us; error 65 us; skew 500 ppm
I20260620 20:10:47.015413 14159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:10:47.017578 14159 webserver.cc:460] Webserver started at http://127.4.169.129:34257/ using document root <none> and password file <none>
I20260620 20:10:47.018275 14159 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:47.018399 14159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:47.018759 14159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:47.021426 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "86469753efd548119cc8dc28e3141e04"
format_stamp: "Formatted at 2026-06-20 20:10:47 on dist-test-slave-4bwq"
server_key: "eec7c8e81899da48ed57a6ed53a1db78"
server_key_iv: "dfcd96c71e84fe1daa15b68c057c17c0"
server_key_version: "encryptionkey@0"
I20260620 20:10:47.022209 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "86469753efd548119cc8dc28e3141e04"
format_stamp: "Formatted at 2026-06-20 20:10:47 on dist-test-slave-4bwq"
server_key: "eec7c8e81899da48ed57a6ed53a1db78"
server_key_iv: "dfcd96c71e84fe1daa15b68c057c17c0"
server_key_version: "encryptionkey@0"
I20260620 20:10:47.027244 14159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:10:47.031694 14173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:47.034037 14159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 20:10:47.034220 14159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "86469753efd548119cc8dc28e3141e04"
format_stamp: "Formatted at 2026-06-20 20:10:47 on dist-test-slave-4bwq"
server_key: "eec7c8e81899da48ed57a6ed53a1db78"
server_key_iv: "dfcd96c71e84fe1daa15b68c057c17c0"
server_key_version: "encryptionkey@0"
I20260620 20:10:47.034444 14159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:47.070765 14159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:47.071667 14159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:47.071969 14159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:47.072964 14159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:47.074482 14159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:47.074595 14159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:47.074723 14159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:47.074821 14159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:47.108371 14159 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:42799
I20260620 20:10:47.108459 14285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:42799 every 8 connection(s)
I20260620 20:10:47.109725 14159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:10:47.113924 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 14159
I20260620 20:10:47.114137 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:10:47.114885 4774 external_mini_cluster.cc:1428] Setting key c4ede2c232b3f062c77d8cc7798bf152
I20260620 20:10:47.117547 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:39961
--builtin_ntp_servers=127.4.169.148:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:10:47.123445 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:47.112290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:42039 (local address 127.4.169.190:39961)
0620 20:10:47.112723 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:47.112729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:47.113933 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 20:10:47.116055 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:47.116060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:47.116147 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:47.116275 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:47.117478 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:47.118274 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:47.119879 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:47.120282 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:47.120423 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:47.121922 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:47.121931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:47.121939 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:47.121963 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:47.122018 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:47.122037 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:47.122133 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:47.122568 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 20:10:47.122777 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":246,"threads_started":1}
I20260620 20:10:47.124308 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:47.112859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:39961 (local address 127.4.169.129:42039)
0620 20:10:47.113780 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:47.113816 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:47.115768 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:47.116553 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:47.116565 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:47.116690 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:47.117316 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:47.117333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:47.118463 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:47.118467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:47.119708 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:47.119733 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:47.120506 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:47.120513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:47.120764 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:47.121497 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:10:47.121500 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:47.121538 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:10:47.121670 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:47.122162 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:47.122176 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:47.122217 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:10:47.122481 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 20:10:47.122904 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":299,"threads_started":1}
I20260620 20:10:47.125881 14286 heartbeater.cc:347] Connected to a master server at 127.4.169.190:39961
I20260620 20:10:47.126174 14286 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:47.126969 14286 heartbeater.cc:511] Master 127.4.169.190:39961 requested a full tablet report, sending...
I20260620 20:10:47.128911 14100 ts_manager.cc:194] Registered new tserver with Master: 86469753efd548119cc8dc28e3141e04 (127.4.169.129:42799)
I20260620 20:10:47.130313 14100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.129:42039
W20260620 20:10:47.441890 14290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:47.442165 14290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:47.442354 14290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:47.449363 14290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:47.449456 14290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:47.449645 14290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:10:47.458258 14290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:47.459528 14290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:47.461480 14290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:47.475551 14298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:47.475584 14295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:47.475494 14296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:47.476574 14290 server_base.cc:1034] running on GCE node
I20260620 20:10:47.477195 14290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:47.478142 14290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:47.479362 14290 hybrid_clock.cc:648] HybridClock initialized: now 1781986247479279 us; error 80 us; skew 500 ppm
I20260620 20:10:47.479681 14290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:10:47.482515 14290 webserver.cc:460] Webserver started at http://127.4.169.130:39083/ using document root <none> and password file <none>
I20260620 20:10:47.483246 14290 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:47.483333 14290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:47.483614 14290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:47.486348 14290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "94a67e17779544c3a3340d4ddcb71a6c"
format_stamp: "Formatted at 2026-06-20 20:10:47 on dist-test-slave-4bwq"
server_key: "b2a03662c98dccb443d4629826d99066"
server_key_iv: "1941de905489bda1a2b12602bbd3ff03"
server_key_version: "encryptionkey@0"
I20260620 20:10:47.487191 14290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "94a67e17779544c3a3340d4ddcb71a6c"
format_stamp: "Formatted at 2026-06-20 20:10:47 on dist-test-slave-4bwq"
server_key: "b2a03662c98dccb443d4629826d99066"
server_key_iv: "1941de905489bda1a2b12602bbd3ff03"
server_key_version: "encryptionkey@0"
I20260620 20:10:47.492568 14290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:10:47.497565 14304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:47.499456 14290 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 20:10:47.499640 14290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "94a67e17779544c3a3340d4ddcb71a6c"
format_stamp: "Formatted at 2026-06-20 20:10:47 on dist-test-slave-4bwq"
server_key: "b2a03662c98dccb443d4629826d99066"
server_key_iv: "1941de905489bda1a2b12602bbd3ff03"
server_key_version: "encryptionkey@0"
I20260620 20:10:47.499897 14290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:47.526262 14290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:47.527408 14290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:47.527729 14290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:47.528780 14290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:47.530519 14290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:47.530644 14290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:47.530756 14290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:47.530824 14290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:47.567795 14290 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:38475
I20260620 20:10:47.567883 14416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:38475 every 8 connection(s)
I20260620 20:10:47.569119 14290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:10:47.570467 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 14290
I20260620 20:10:47.570655 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:10:47.571228 4774 external_mini_cluster.cc:1428] Setting key 988a1c48e3a7e69e69fe48b20cf3ba4c
I20260620 20:10:47.573978 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:39961
--builtin_ntp_servers=127.4.169.148:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:10:47.582526 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:47.571896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:43809 (local address 127.4.169.190:39961)
0620 20:10:47.572071 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:47.572077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:47.573730 (+ 1653us) server_negotiation.cc:407] Connection header received
0620 20:10:47.575613 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:47.575620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:47.575711 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:47.575830 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:47.576783 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:47.577897 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:47.579437 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:47.580173 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:47.580313 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:47.581815 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:47.581820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:47.581823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:47.581839 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:47.581867 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:47.581878 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:47.581941 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:47.582179 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 20:10:47.582251 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:10:47.583926 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:47.572528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:39961 (local address 127.4.169.130:43809)
0620 20:10:47.573563 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:47.573606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:47.575361 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:47.576001 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:47.576009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:47.576091 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:47.576623 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:47.576637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:47.578031 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:47.578035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:47.579301 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:47.579313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:47.580412 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:47.580421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:47.580745 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:47.581466 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:10:47.581472 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:47.581532 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:10:47.581630 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:47.581967 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:47.581979 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:47.581989 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:10:47.582162 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 20:10:47.582595 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":340,"threads_started":1}
I20260620 20:10:47.585249 14417 heartbeater.cc:347] Connected to a master server at 127.4.169.190:39961
I20260620 20:10:47.585531 14417 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:47.586252 14417 heartbeater.cc:511] Master 127.4.169.190:39961 requested a full tablet report, sending...
I20260620 20:10:47.587648 14100 ts_manager.cc:194] Registered new tserver with Master: 94a67e17779544c3a3340d4ddcb71a6c (127.4.169.130:38475)
I20260620 20:10:47.588429 14100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.130:43809
W20260620 20:10:47.900432 14420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:47.900740 14420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:47.900900 14420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:47.908104 14420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:47.908232 14420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:47.908404 14420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:10:47.917380 14420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:47.918659 14420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:47.920360 14420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:47.932746 14425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:47.932786 14426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:47.932824 14428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:47.933702 14420 server_base.cc:1034] running on GCE node
I20260620 20:10:47.934341 14420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:47.935266 14420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:47.936455 14420 hybrid_clock.cc:648] HybridClock initialized: now 1781986247936416 us; error 35 us; skew 500 ppm
I20260620 20:10:47.936762 14420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:10:47.939014 14420 webserver.cc:460] Webserver started at http://127.4.169.131:36343/ using document root <none> and password file <none>
I20260620 20:10:47.939780 14420 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:47.939893 14420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:47.940255 14420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:47.942989 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "8f8e6b2a63da465da2ca4543142f17d0"
format_stamp: "Formatted at 2026-06-20 20:10:47 on dist-test-slave-4bwq"
server_key: "ddaa3239febc7912d16973191b46e381"
server_key_iv: "1d1d6ca39d4f5d60ded07012de977a5e"
server_key_version: "encryptionkey@0"
I20260620 20:10:47.943756 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "8f8e6b2a63da465da2ca4543142f17d0"
format_stamp: "Formatted at 2026-06-20 20:10:47 on dist-test-slave-4bwq"
server_key: "ddaa3239febc7912d16973191b46e381"
server_key_iv: "1d1d6ca39d4f5d60ded07012de977a5e"
server_key_version: "encryptionkey@0"
I20260620 20:10:47.948858 14420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:10:47.952435 14434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:47.953871 14420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:47.954051 14420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "8f8e6b2a63da465da2ca4543142f17d0"
format_stamp: "Formatted at 2026-06-20 20:10:47 on dist-test-slave-4bwq"
server_key: "ddaa3239febc7912d16973191b46e381"
server_key_iv: "1d1d6ca39d4f5d60ded07012de977a5e"
server_key_version: "encryptionkey@0"
I20260620 20:10:47.954286 14420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:47.967999 14420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:47.968918 14420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:47.969247 14420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:47.970244 14420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:47.971740 14420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:47.971851 14420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:47.971995 14420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:47.972079 14420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:48.004742 14420 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:46303
I20260620 20:10:48.004797 14546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:46303 every 8 connection(s)
I20260620 20:10:48.006163 14420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:10:48.007651 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 14420
I20260620 20:10:48.007836 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:10:48.008361 4774 external_mini_cluster.cc:1428] Setting key f7801813d4965338fb435933316cc9ab
I20260620 20:10:48.018282 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:48.008853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:54419 (local address 127.4.169.190:39961)
0620 20:10:48.009079 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:48.009088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:48.010703 (+ 1615us) server_negotiation.cc:407] Connection header received
0620 20:10:48.012557 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:48.012561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:48.012623 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:48.012705 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:48.013711 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.014485 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.016024 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.016428 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.016523 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:48.017669 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:48.017674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:48.017677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:48.017692 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:48.017720 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:48.017731 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:48.017796 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:48.017980 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 20:10:48.018045 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:10:48.019456 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:48.009476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:39961 (local address 127.4.169.131:54419)
0620 20:10:48.010544 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:48.010599 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:48.012329 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:48.012903 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:48.012910 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:48.013000 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:48.013578 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:48.013593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.014612 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.014616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:48.015828 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:48.015839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.016600 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.016604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:48.016796 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:48.017403 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:10:48.017406 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:48.017446 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:10:48.017514 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:48.017812 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:48.017819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:48.017824 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:10:48.017961 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 20:10:48.018230 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":292,"threads_started":1}
I20260620 20:10:48.020740 14547 heartbeater.cc:347] Connected to a master server at 127.4.169.190:39961
I20260620 20:10:48.021039 14547 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:48.021761 14547 heartbeater.cc:511] Master 127.4.169.190:39961 requested a full tablet report, sending...
I20260620 20:10:48.023343 14100 ts_manager.cc:194] Registered new tserver with Master: 8f8e6b2a63da465da2ca4543142f17d0 (127.4.169.131:46303)
I20260620 20:10:48.023912 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:10:48.024247 14100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.131:54419
W20260620 20:10:48.030436 14288 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:48.028952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.4.169.190:39961)
0620 20:10:48.029124 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:48.029129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:48.029606 (+ 477us) server_negotiation.cc:407] Connection header received
0620 20:10:48.029729 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:48.029733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:48.029797 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:48.029874 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:48.030181 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58146: BlockingRecv error: recv got EOF from 127.0.0.1:58146 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:10:48.031208 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 14159
I20260620 20:10:48.038259 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:48.032140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.4.169.129:42799)
0620 20:10:48.032666 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:48.032672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:48.032696 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:10:48.032751 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:48.032756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:48.032930 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:48.033051 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:48.033336 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.034258 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.035484 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.035994 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.036087 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:48.036996 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:48.037003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:48.037014 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:48.037557 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:48.037598 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:48.037616 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:48.037698 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:48.037795 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 20:10:48.037924 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":282,"threads_started":1}
I20260620 20:10:48.039381 14281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:48.120268 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 14159
I20260620 20:10:48.136380 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 14290
I20260620 20:10:48.144078 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:48.137292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.4.169.130:38475)
0620 20:10:48.137843 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:48.137849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:48.137876 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:10:48.137939 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:48.137945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:48.138189 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:48.138321 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:48.138705 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.139691 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.140862 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.141416 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.141508 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:48.142631 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:48.142640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:48.142654 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:48.143274 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:48.143319 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:48.143340 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:48.143435 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:48.143578 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 20:10:48.143741 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":294,"threads_started":1}
I20260620 20:10:48.145282 14412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:48.226255 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 14290
I20260620 20:10:48.243000 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 14420
I20260620 20:10:48.251330 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:48.244128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.4.169.131:46303)
0620 20:10:48.244723 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:48.244730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:48.244757 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:10:48.244814 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:48.244820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:48.245032 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:48.245247 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:48.245660 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.246781 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.248013 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:48.248583 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:48.248723 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:48.249786 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:48.249795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:48.249809 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:48.250451 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:48.250513 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:48.250540 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:48.250652 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:48.250886 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 20:10:48.251012 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":317,"threads_started":1}
I20260620 20:10:48.252597 14542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:48.333145 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 14420
I20260620 20:10:48.348438 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 14069
I20260620 20:10:48.349545 14130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:48.449518 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 14069
2026-06-20T20:10:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2397 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T20:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:10:48Z Disabled control of system clock
I20260620 20:10:48.519140 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:46439
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:10:48.846009 14566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:48.846299 14566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:48.846509 14566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:48.853612 14566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:48.853713 14566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:48.853750 14566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:48.853826 14566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:48.853892 14566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:48.862916 14566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:46439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:48.864185 14566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:48.865928 14566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:48.878049 14572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:48.878131 14571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:48.878131 14574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:48.878794 14566 server_base.cc:1034] running on GCE node
I20260620 20:10:48.879566 14566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:48.880990 14566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:48.882225 14566 hybrid_clock.cc:648] HybridClock initialized: now 1781986248882173 us; error 48 us; skew 500 ppm
I20260620 20:10:48.882632 14566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:10:48.884918 14566 webserver.cc:460] Webserver started at http://127.4.169.190:43547/ using document root <none> and password file <none>
I20260620 20:10:48.885607 14566 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:48.885735 14566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:48.886044 14566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:48.890220 14566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "b66fd0656cf348108999b2faa846f1ca"
format_stamp: "Formatted at 2026-06-20 20:10:48 on dist-test-slave-4bwq"
server_key: "5c5a7edc5f259ed6153541726af06532"
server_key_iv: "1686be827165bd41279e1c1e68c77eb4"
server_key_version: "encryptionkey@0"
I20260620 20:10:48.891049 14566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "b66fd0656cf348108999b2faa846f1ca"
format_stamp: "Formatted at 2026-06-20 20:10:48 on dist-test-slave-4bwq"
server_key: "5c5a7edc5f259ed6153541726af06532"
server_key_iv: "1686be827165bd41279e1c1e68c77eb4"
server_key_version: "encryptionkey@0"
I20260620 20:10:48.895854 14566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:10:48.899150 14580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:48.900406 14566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:48.900578 14566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "b66fd0656cf348108999b2faa846f1ca"
format_stamp: "Formatted at 2026-06-20 20:10:48 on dist-test-slave-4bwq"
server_key: "5c5a7edc5f259ed6153541726af06532"
server_key_iv: "1686be827165bd41279e1c1e68c77eb4"
server_key_version: "encryptionkey@0"
I20260620 20:10:48.900782 14566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:48.906512 14566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:48.907275 14566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:48.907577 14566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:48.925442 14566 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:46439
I20260620 20:10:48.925474 14631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:46439 every 8 connection(s)
I20260620 20:10:48.926829 14566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:10:48.929093 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 14566
I20260620 20:10:48.929324 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:10:48.929975 4774 external_mini_cluster.cc:1428] Setting key 767054f6750fb4fc3f1f6b5840da4f18
I20260620 20:10:48.931221 14632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:48.939069 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:48.932046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.4.169.190:46439)
0620 20:10:48.933129 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:48.933152 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:48.933202 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:10:48.934850 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:48.934878 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:48.935468 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:48.935910 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:48.936976 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:48.936985 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:48.937002 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:48.937019 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:48.937110 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:48.937136 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:48.937370 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:48.937504 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 20:10:48.937792 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":327,"threads_started":1}
I20260620 20:10:48.945252 14632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca: Bootstrap starting.
I20260620 20:10:48.948541 14632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:48.949542 14632 log.cc:824] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:48.952358 14632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca: No bootstrap required, opened a new log
I20260620 20:10:48.960120 14632 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66fd0656cf348108999b2faa846f1ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 46439 } }
I20260620 20:10:48.960482 14632 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:48.960557 14632 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66fd0656cf348108999b2faa846f1ca, State: Initialized, Role: FOLLOWER
I20260620 20:10:48.961066 14632 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66fd0656cf348108999b2faa846f1ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 46439 } }
I20260620 20:10:48.961259 14632 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:10:48.961354 14632 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:10:48.961483 14632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:48.963661 14632 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66fd0656cf348108999b2faa846f1ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 46439 } }
I20260620 20:10:48.964138 14632 leader_election.cc:302] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66fd0656cf348108999b2faa846f1ca; no voters:
I20260620 20:10:48.964663 14632 leader_election.cc:288] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:10:48.964830 14637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:48.965572 14637 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [term 1 LEADER]: Becoming Leader. State: Replica: b66fd0656cf348108999b2faa846f1ca, State: Running, Role: LEADER
I20260620 20:10:48.966181 14637 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66fd0656cf348108999b2faa846f1ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 46439 } }
I20260620 20:10:48.966929 14632 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:48.969167 14638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66fd0656cf348108999b2faa846f1ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66fd0656cf348108999b2faa846f1ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 46439 } } }
I20260620 20:10:48.969542 14638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:48.969292 14639 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66fd0656cf348108999b2faa846f1ca. Latest consensus state: current_term: 1 leader_uuid: "b66fd0656cf348108999b2faa846f1ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66fd0656cf348108999b2faa846f1ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 46439 } } }
I20260620 20:10:48.969712 14639 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:48.972628 14643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:48.978118 14643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:48.986799 14643 catalog_manager.cc:1347] Generated new cluster ID: b4b0ee0a04c84e75828157757bd47faf
I20260620 20:10:48.986922 14643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:49.010365 14643 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:10:49.010876 14643 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:49.019330 14643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66fd0656cf348108999b2faa846f1ca: Generated new TSK 0
I20260620 20:10:49.020195 14643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:10:49.025463 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:46439
--builtin_ntp_servers=127.4.169.148:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:10:49.356308 14656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:49.356575 14656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:49.356745 14656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:49.363868 14656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:49.363961 14656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:49.364148 14656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:10:49.373344 14656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:49.374583 14656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:49.376289 14656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:49.388845 14661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:49.388809 14664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:49.388775 14662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:49.389693 14656 server_base.cc:1034] running on GCE node
I20260620 20:10:49.390285 14656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:49.391263 14656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:49.392473 14656 hybrid_clock.cc:648] HybridClock initialized: now 1781986249392421 us; error 40 us; skew 500 ppm
I20260620 20:10:49.392789 14656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:10:49.395048 14656 webserver.cc:460] Webserver started at http://127.4.169.129:45741/ using document root <none> and password file <none>
I20260620 20:10:49.395716 14656 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:49.395824 14656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:49.396148 14656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:49.398958 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "334b5ab9b85f4c82a0f01e7b0a190f2f"
format_stamp: "Formatted at 2026-06-20 20:10:49 on dist-test-slave-4bwq"
server_key: "79eeb075e7b2617c04f4529b4b68c3cc"
server_key_iv: "502dd4b92bb70a4a2405a8f1c3e08494"
server_key_version: "encryptionkey@0"
I20260620 20:10:49.399791 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "334b5ab9b85f4c82a0f01e7b0a190f2f"
format_stamp: "Formatted at 2026-06-20 20:10:49 on dist-test-slave-4bwq"
server_key: "79eeb075e7b2617c04f4529b4b68c3cc"
server_key_iv: "502dd4b92bb70a4a2405a8f1c3e08494"
server_key_version: "encryptionkey@0"
I20260620 20:10:49.404826 14656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:10:49.408458 14670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:49.409799 14656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:49.409984 14656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "334b5ab9b85f4c82a0f01e7b0a190f2f"
format_stamp: "Formatted at 2026-06-20 20:10:49 on dist-test-slave-4bwq"
server_key: "79eeb075e7b2617c04f4529b4b68c3cc"
server_key_iv: "502dd4b92bb70a4a2405a8f1c3e08494"
server_key_version: "encryptionkey@0"
I20260620 20:10:49.410763 14656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:49.417622 14656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:49.418522 14656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:49.418802 14656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:49.419718 14656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:49.421109 14656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:49.421201 14656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:49.421347 14656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:49.421444 14656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:49.454671 14656 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:44799
I20260620 20:10:49.454730 14782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:44799 every 8 connection(s)
I20260620 20:10:49.456035 14656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:10:49.460969 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 14656
I20260620 20:10:49.461161 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:10:49.461838 4774 external_mini_cluster.cc:1428] Setting key 53c49a5fcd984b562ede78b16142e9e6
I20260620 20:10:49.464293 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:49.458555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:48193 (local address 127.4.169.190:46439)
0620 20:10:49.459022 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:49.459027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:49.460128 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 20:10:49.461914 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:49.461920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:49.461986 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:49.462116 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:49.463600 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:49.463606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:49.463611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:49.463629 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:49.463679 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:49.463695 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:49.463817 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:49.464031 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 20:10:49.464077 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260620 20:10:49.464911 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:46439
--builtin_ntp_servers=127.4.169.148:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 20:10:49.465564 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:49.459080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:46439 (local address 127.4.169.129:48193)
0620 20:10:49.459970 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:49.460006 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:49.461639 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:49.462361 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:49.462370 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:49.462534 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:49.463271 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:10:49.463273 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:49.463311 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:10:49.463394 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:49.463818 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:49.463828 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:49.463835 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:10:49.464110 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 20:10:49.464363 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":260,"threads_started":1}
I20260620 20:10:49.467159 14783 heartbeater.cc:347] Connected to a master server at 127.4.169.190:46439
I20260620 20:10:49.467473 14783 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:49.468093 14783 heartbeater.cc:511] Master 127.4.169.190:46439 requested a full tablet report, sending...
I20260620 20:10:49.470126 14597 ts_manager.cc:194] Registered new tserver with Master: 334b5ab9b85f4c82a0f01e7b0a190f2f (127.4.169.129:44799)
W20260620 20:10:49.791968 14787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:49.792394 14787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:49.792554 14787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:49.803807 14787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:49.803946 14787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:49.804248 14787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:10:49.814256 14787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:49.815675 14787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:49.817488 14787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:49.830269 14793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:49.830302 14795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:49.830340 14792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:49.831039 14787 server_base.cc:1034] running on GCE node
I20260620 20:10:49.831576 14787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:49.832451 14787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:49.833642 14787 hybrid_clock.cc:648] HybridClock initialized: now 1781986249833598 us; error 41 us; skew 500 ppm
I20260620 20:10:49.833959 14787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:10:49.836141 14787 webserver.cc:460] Webserver started at http://127.4.169.130:36109/ using document root <none> and password file <none>
I20260620 20:10:49.836849 14787 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:49.836933 14787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:49.837208 14787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:49.839928 14787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "9535b2f8238b448e93a9d8110cb776ce"
format_stamp: "Formatted at 2026-06-20 20:10:49 on dist-test-slave-4bwq"
server_key: "367ed4a066d82b7aae7398172469e562"
server_key_iv: "fef791bceba46362e715b0dbae7cad0b"
server_key_version: "encryptionkey@0"
I20260620 20:10:49.840621 14787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "9535b2f8238b448e93a9d8110cb776ce"
format_stamp: "Formatted at 2026-06-20 20:10:49 on dist-test-slave-4bwq"
server_key: "367ed4a066d82b7aae7398172469e562"
server_key_iv: "fef791bceba46362e715b0dbae7cad0b"
server_key_version: "encryptionkey@0"
I20260620 20:10:49.845525 14787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:10:49.848857 14801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:49.850108 14787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:10:49.850231 14787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "9535b2f8238b448e93a9d8110cb776ce"
format_stamp: "Formatted at 2026-06-20 20:10:49 on dist-test-slave-4bwq"
server_key: "367ed4a066d82b7aae7398172469e562"
server_key_iv: "fef791bceba46362e715b0dbae7cad0b"
server_key_version: "encryptionkey@0"
I20260620 20:10:49.850476 14787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:49.855659 14787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:49.856343 14787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:49.856559 14787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:49.857420 14787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:49.858748 14787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:49.858830 14787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:49.858906 14787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:49.858958 14787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:49.891913 14787 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:34597
I20260620 20:10:49.891965 14913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:34597 every 8 connection(s)
I20260620 20:10:49.893252 14787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:10:49.897712 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 14787
I20260620 20:10:49.897871 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:10:49.898324 4774 external_mini_cluster.cc:1428] Setting key 1c54fe8a4cf201508459b23d0e43cf48
I20260620 20:10:49.901464 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:46439
--builtin_ntp_servers=127.4.169.148:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 20:10:49.901993 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:49.895799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:55059 (local address 127.4.169.190:46439)
0620 20:10:49.895978 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:49.895982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:49.897418 (+ 1436us) server_negotiation.cc:407] Connection header received
0620 20:10:49.899443 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:49.899447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:49.899512 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:49.899603 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:49.901146 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:49.901155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:49.901163 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:49.901187 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:49.901233 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:49.901253 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:49.901388 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:49.901575 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 20:10:49.901641 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:10:49.902916 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:49.896346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:46439 (local address 127.4.169.130:55059)
0620 20:10:49.897238 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:49.897289 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:49.899164 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:49.899872 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:49.899884 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:49.900039 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:49.900800 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:10:49.900803 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:49.900842 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:10:49.900925 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:49.901359 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:49.901366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:49.901372 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:10:49.901606 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 20:10:49.901891 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":274,"threads_started":1}
I20260620 20:10:49.904326 14914 heartbeater.cc:347] Connected to a master server at 127.4.169.190:46439
I20260620 20:10:49.904618 14914 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:49.905205 14914 heartbeater.cc:511] Master 127.4.169.190:46439 requested a full tablet report, sending...
I20260620 20:10:49.906720 14597 ts_manager.cc:194] Registered new tserver with Master: 9535b2f8238b448e93a9d8110cb776ce (127.4.169.130:34597)
W20260620 20:10:50.234839 14917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:50.235117 14917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:50.235303 14917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:50.242534 14917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:50.242635 14917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:50.242832 14917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:10:50.251844 14917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:50.253226 14917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:50.255075 14917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:50.268702 14923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:50.268762 14925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:50.268788 14922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:50.269668 14917 server_base.cc:1034] running on GCE node
I20260620 20:10:50.270227 14917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:50.271224 14917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:50.272420 14917 hybrid_clock.cc:648] HybridClock initialized: now 1781986250272328 us; error 89 us; skew 500 ppm
I20260620 20:10:50.272802 14917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:10:50.280045 14917 webserver.cc:460] Webserver started at http://127.4.169.131:35445/ using document root <none> and password file <none>
I20260620 20:10:50.280861 14917 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:50.280994 14917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:50.281375 14917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:50.284564 14917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "9083255deec348128a565e3c20a3aec3"
format_stamp: "Formatted at 2026-06-20 20:10:50 on dist-test-slave-4bwq"
server_key: "401552c87a4c2d54b5a4c9d25e27d00b"
server_key_iv: "56d8473cae9b64fc7d80c51155dc6223"
server_key_version: "encryptionkey@0"
I20260620 20:10:50.285379 14917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "9083255deec348128a565e3c20a3aec3"
format_stamp: "Formatted at 2026-06-20 20:10:50 on dist-test-slave-4bwq"
server_key: "401552c87a4c2d54b5a4c9d25e27d00b"
server_key_iv: "56d8473cae9b64fc7d80c51155dc6223"
server_key_version: "encryptionkey@0"
I20260620 20:10:50.290730 14917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:10:50.294090 14931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:50.295501 14917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:50.295668 14917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "9083255deec348128a565e3c20a3aec3"
format_stamp: "Formatted at 2026-06-20 20:10:50 on dist-test-slave-4bwq"
server_key: "401552c87a4c2d54b5a4c9d25e27d00b"
server_key_iv: "56d8473cae9b64fc7d80c51155dc6223"
server_key_version: "encryptionkey@0"
I20260620 20:10:50.295898 14917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:50.301694 14917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:50.302528 14917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:50.302793 14917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:50.303730 14917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:50.305217 14917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:50.305326 14917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:50.305451 14917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:50.305570 14917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 20:10:50.340265 14917 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:42947
I20260620 20:10:50.340365 15043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:42947 every 8 connection(s)
I20260620 20:10:50.341682 14917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:10:50.345353 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 14917
I20260620 20:10:50.345520 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:10:50.346074 4774 external_mini_cluster.cc:1428] Setting key 6a3f78e25066077e9f8ee3f8740dfa21
I20260620 20:10:50.350860 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:50.344309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:60145 (local address 127.4.169.190:46439)
0620 20:10:50.344475 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:50.344479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:50.346148 (+ 1669us) server_negotiation.cc:407] Connection header received
0620 20:10:50.348512 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:50.348517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:50.348583 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:50.348694 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:50.350150 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:50.350155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:50.350158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:50.350173 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:50.350205 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:50.350216 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:50.350314 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:50.350602 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 20:10:50.350652 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:10:50.351781 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:50.344796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:46439 (local address 127.4.169.131:60145)
0620 20:10:50.345952 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:50.345998 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:50.348234 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:50.348899 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:50.348934 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:50.349077 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:50.349840 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:10:50.349843 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:50.349884 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:10:50.349975 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:50.350302 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:50.350308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:50.350314 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:10:50.350595 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 20:10:50.350811 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":325,"threads_started":1}
I20260620 20:10:50.353129 15044 heartbeater.cc:347] Connected to a master server at 127.4.169.190:46439
I20260620 20:10:50.353461 15044 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:50.354135 15044 heartbeater.cc:511] Master 127.4.169.190:46439 requested a full tablet report, sending...
I20260620 20:10:50.355697 14597 ts_manager.cc:194] Registered new tserver with Master: 9083255deec348128a565e3c20a3aec3 (127.4.169.131:42947)
I20260620 20:10:50.361526 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:10:50.367878 14785 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:10:50.366085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.4.169.190:46439)
0620 20:10:50.366255 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:50.366260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:50.366848 (+ 588us) server_negotiation.cc:407] Connection header received
0620 20:10:50.367081 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:50.367086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:50.367140 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:50.367239 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:50.367599 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41214: BlockingRecv error: recv got EOF from 127.0.0.1:41214 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:10:50.368253 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 14656
I20260620 20:10:50.372107 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:50.369292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.4.169.129:44799)
0620 20:10:50.369889 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:50.369898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:50.369926 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:10:50.369983 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:50.369990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:50.370211 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:50.370346 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:50.371302 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:50.371309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:50.371321 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:50.371339 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:50.371387 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:50.371406 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:50.371528 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:50.371719 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 20:10:50.371804 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":300,"threads_started":1}
I20260620 20:10:50.373473 14778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:50.452900 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 14656
I20260620 20:10:50.469158 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 14787
I20260620 20:10:50.473189 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:50.470220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.4.169.130:34597)
0620 20:10:50.470740 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:50.470747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:50.470772 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:10:50.470833 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:50.470839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:50.471095 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:50.471272 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:50.472404 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:50.472410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:50.472423 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:50.472440 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:50.472481 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:50.472494 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:50.472601 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:50.472758 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 20:10:50.472860 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":268,"threads_started":1}
I20260620 20:10:50.474517 14909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:50.555681 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 14787
I20260620 20:10:50.571040 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 14917
I20260620 20:10:50.575037 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:50.572142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.4.169.131:42947)
0620 20:10:50.572868 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:50.572875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:50.572905 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:10:50.572966 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:50.572973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:50.573211 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:50.573342 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:50.574314 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:50.574320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:50.574333 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:10:50.574348 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:50.574452 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:10:50.574471 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:50.574615 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:50.574671 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 20:10:50.574751 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":406,"threads_started":1}
I20260620 20:10:50.576349 15039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:50.657657 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 14917
I20260620 20:10:50.674257 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 14566
I20260620 20:10:50.675645 14627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:50.775794 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 14566
2026-06-20T20:10:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2326 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:10:50 dist-test-slave-4bwq krb5kdc[15062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:10:50 dist-test-slave-4bwq krb5kdc[15062](info): set up 2 sockets
Jun 20 20:10:50 dist-test-slave-4bwq krb5kdc[15062](info): commencing operation
krb5kdc: starting...
W20260620 20:10:51.871506 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:10:52 dist-test-slave-4bwq krb5kdc[15062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:10:52.089597 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:33031
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:33031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:52.410501 15075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:52.410826 15075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:52.411005 15075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:52.418417 15075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:10:52.418534 15075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:52.418673 15075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:52.418767 15075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:10:52.418851 15075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:10:52.428992 15075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:33031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:33031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:52.430505 15075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:52.431807 15075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:52.443614 15080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:52.443575 15083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:52.443591 15081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:52.444430 15075 server_base.cc:1034] running on GCE node
I20260620 20:10:52.445202 15075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:52.446684 15075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:52.447891 15075 hybrid_clock.cc:648] HybridClock initialized: now 1781986252447845 us; error 32 us; skew 500 ppm
I20260620 20:10:52.448238 15075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:52 dist-test-slave-4bwq krb5kdc[15062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986252, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:52.451428 15075 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:10:52.452939 15075 webserver.cc:460] Webserver started at http://127.4.169.190:40853/ using document root <none> and password file <none>
I20260620 20:10:52.453626 15075 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:52.453771 15075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:52.454061 15075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:52.457304 15075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "daab4aab6d7142dc8912a602207a9cb8"
format_stamp: "Formatted at 2026-06-20 20:10:52 on dist-test-slave-4bwq"
server_key: "0af659014b75a2d694440e60bf16ef63"
server_key_iv: "84791a5533e217023ab0c7a166dc0cc7"
server_key_version: "encryptionkey@0"
I20260620 20:10:52.458120 15075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "daab4aab6d7142dc8912a602207a9cb8"
format_stamp: "Formatted at 2026-06-20 20:10:52 on dist-test-slave-4bwq"
server_key: "0af659014b75a2d694440e60bf16ef63"
server_key_iv: "84791a5533e217023ab0c7a166dc0cc7"
server_key_version: "encryptionkey@0"
I20260620 20:10:52.463511 15075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:10:52.467217 15090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:52.468672 15075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:10:52.468873 15075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "daab4aab6d7142dc8912a602207a9cb8"
format_stamp: "Formatted at 2026-06-20 20:10:52 on dist-test-slave-4bwq"
server_key: "0af659014b75a2d694440e60bf16ef63"
server_key_iv: "84791a5533e217023ab0c7a166dc0cc7"
server_key_version: "encryptionkey@0"
I20260620 20:10:52.469125 15075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:52.504487 15075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:52.508994 15075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:52.509300 15075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:52.528635 15075 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:33031
I20260620 20:10:52.528678 15141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:33031 every 8 connection(s)
I20260620 20:10:52.530041 15075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:10:52.531770 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 15075
I20260620 20:10:52.531966 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:10:52.532619 4774 external_mini_cluster.cc:1428] Setting key 20dc732b615f88fcbe6e244a953cc549
I20260620 20:10:52.534430 15142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 20:10:52 dist-test-slave-4bwq krb5kdc[15062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:52.548156 15142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8: Bootstrap starting.
I20260620 20:10:52.551278 15142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:52.552302 15142 log.cc:824] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:52.555218 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:52.534749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.4.169.190:33031)
0620 20:10:52.535541 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:52.535552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:52.535591 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:10:52.537075 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:52.537095 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:52.537487 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:52.537822 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:52.539692 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:52.541014 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:52.542773 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:52.543427 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:52.543557 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:52.548103 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:52.548111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:52.548125 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:52.548167 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:52.551272 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:52.551979 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:52.551985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:52.551992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:52.552177 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:52.552580 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:52.552584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:52.552586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:52.553091 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:52.553437 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:52.553789 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 20:10:52.554211 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":366,"threads_started":1}
I20260620 20:10:52.555299 15142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8: No bootstrap required, opened a new log
I20260620 20:10:52.563112 15142 raft_consensus.cc:348] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daab4aab6d7142dc8912a602207a9cb8" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 33031 } }
I20260620 20:10:52.563535 15142 raft_consensus.cc:374] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:52.563649 15142 raft_consensus.cc:729] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daab4aab6d7142dc8912a602207a9cb8, State: Initialized, Role: FOLLOWER
I20260620 20:10:52.564188 15142 consensus_queue.cc:260] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daab4aab6d7142dc8912a602207a9cb8" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 33031 } }
I20260620 20:10:52.564433 15142 raft_consensus.cc:388] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:10:52.564574 15142 raft_consensus.cc:482] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:10:52.564757 15142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:52.567203 15142 raft_consensus.cc:504] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daab4aab6d7142dc8912a602207a9cb8" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 33031 } }
I20260620 20:10:52.567703 15142 leader_election.cc:302] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daab4aab6d7142dc8912a602207a9cb8; no voters:
I20260620 20:10:52.568281 15142 leader_election.cc:288] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:10:52.568414 15147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:52.569108 15147 raft_consensus.cc:686] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [term 1 LEADER]: Becoming Leader. State: Replica: daab4aab6d7142dc8912a602207a9cb8, State: Running, Role: LEADER
I20260620 20:10:52.569638 15147 consensus_queue.cc:237] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daab4aab6d7142dc8912a602207a9cb8" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 33031 } }
I20260620 20:10:52.570363 15142 sys_catalog.cc:564] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:10:52.572196 15148 sys_catalog.cc:455] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daab4aab6d7142dc8912a602207a9cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daab4aab6d7142dc8912a602207a9cb8" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 33031 } } }
I20260620 20:10:52.572894 15148 sys_catalog.cc:458] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:52.572561 15149 sys_catalog.cc:455] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daab4aab6d7142dc8912a602207a9cb8. Latest consensus state: current_term: 1 leader_uuid: "daab4aab6d7142dc8912a602207a9cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daab4aab6d7142dc8912a602207a9cb8" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 33031 } } }
I20260620 20:10:52.574038 15153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:10:52.574098 15149 sys_catalog.cc:458] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:10:52.580646 15153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:10:52.589526 15153 catalog_manager.cc:1347] Generated new cluster ID: 6afa23d362c44820a280c5199f0dc575
I20260620 20:10:52.589639 15153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:10:52.610486 15153 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:10:52.611850 15153 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:10:52.621371 15153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8: Generated new TSK 0
I20260620 20:10:52.622295 15153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:10:52.688372 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:33031
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:53.021773 15168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:53.022110 15168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:53.022286 15168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:53.030865 15168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:53.031032 15168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:53.031276 15168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:10:53.043180 15168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:33031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:53.044450 15168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:53.045817 15168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:53.059494 15174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:53.059532 15173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:53.059446 15176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:53.060276 15168 server_base.cc:1034] running on GCE node
I20260620 20:10:53.060874 15168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:53.061837 15168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:53.063042 15168 hybrid_clock.cc:648] HybridClock initialized: now 1781986253062994 us; error 35 us; skew 500 ppm
I20260620 20:10:53.063393 15168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:53 dist-test-slave-4bwq krb5kdc[15062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986253, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:53.066727 15168 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:10:53.068269 15168 webserver.cc:460] Webserver started at http://127.4.169.129:44283/ using document root <none> and password file <none>
I20260620 20:10:53.068974 15168 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:53.069108 15168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:53.069422 15168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:53.072652 15168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "e3e621f546f64108bea9a940eded17fc"
format_stamp: "Formatted at 2026-06-20 20:10:53 on dist-test-slave-4bwq"
server_key: "9b881d90882077fea1e2cd1a6a5603f4"
server_key_iv: "cd1c9f0fc6e034fc61997824acf7deb2"
server_key_version: "encryptionkey@0"
I20260620 20:10:53.073491 15168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "e3e621f546f64108bea9a940eded17fc"
format_stamp: "Formatted at 2026-06-20 20:10:53 on dist-test-slave-4bwq"
server_key: "9b881d90882077fea1e2cd1a6a5603f4"
server_key_iv: "cd1c9f0fc6e034fc61997824acf7deb2"
server_key_version: "encryptionkey@0"
I20260620 20:10:53.079324 15168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:10:53.082930 15183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:53.084371 15168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:10:53.084550 15168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "e3e621f546f64108bea9a940eded17fc"
format_stamp: "Formatted at 2026-06-20 20:10:53 on dist-test-slave-4bwq"
server_key: "9b881d90882077fea1e2cd1a6a5603f4"
server_key_iv: "cd1c9f0fc6e034fc61997824acf7deb2"
server_key_version: "encryptionkey@0"
I20260620 20:10:53.084745 15168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:53.107424 15168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:53.111373 15168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:53.111656 15168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:53.112675 15168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:53.113961 15168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:53.114035 15168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:53.114203 15168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:53.114264 15168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:53.147913 15168 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:41765
I20260620 20:10:53.147957 15295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:41765 every 8 connection(s)
I20260620 20:10:53.149281 15168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:10:53.153447 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 15168
I20260620 20:10:53.153676 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:10:53.154304 4774 external_mini_cluster.cc:1428] Setting key b1a237baa20a5dd48bc8e730407c29de
Jun 20 20:10:53 dist-test-slave-4bwq krb5kdc[15062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986253, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:53.169735 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:53.151859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:33507 (local address 127.4.169.190:33031)
0620 20:10:53.152369 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:53.152374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:53.153203 (+ 829us) server_negotiation.cc:407] Connection header received
0620 20:10:53.155276 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:53.155282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:53.155366 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:53.155492 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:53.157039 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:53.157748 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:53.159387 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:53.159786 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:53.159923 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:53.163511 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:53.163516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:53.163519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:53.163553 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:53.166182 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:53.167273 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:53.167278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:53.167281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:53.167413 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:53.167953 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:53.167957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:53.167959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:53.168228 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:53.168339 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:53.169298 (+ 959us) server_negotiation.cc:299] Negotiation successful
0620 20:10:53.169471 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":298,"threads_started":1}
I20260620 20:10:53.171412 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:53.152299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:33031 (local address 127.4.169.129:33507)
0620 20:10:53.153037 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:53.153076 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:53.154982 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:53.155751 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:53.155760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:53.156276 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:53.156865 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:53.156882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:53.157904 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:53.157909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:53.159230 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:53.159243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:53.160013 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:53.160023 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:53.160263 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:53.160968 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:53.160991 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:53.163264 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:53.166351 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:53.166363 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:53.166428 (+ 65us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:53.167105 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:53.167533 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:53.167540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:53.167547 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:53.167815 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:53.168359 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:53.168370 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:53.168869 (+ 499us) client_negotiation.cc:769] Sending connection context
0620 20:10:53.169202 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 20:10:53.169835 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":274,"threads_started":1}
I20260620 20:10:53.173175 15296 heartbeater.cc:347] Connected to a master server at 127.4.169.190:33031
I20260620 20:10:53.173491 15296 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:53.174230 15296 heartbeater.cc:511] Master 127.4.169.190:33031 requested a full tablet report, sending...
I20260620 20:10:53.176460 15107 ts_manager.cc:194] Registered new tserver with Master: e3e621f546f64108bea9a940eded17fc (127.4.169.129:41765)
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:10:53.177999 15107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:33507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:10:53.207566 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:33031
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:53.536034 15302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:53.536295 15302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:53.536453 15302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:53.543725 15302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:53.543833 15302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:53.544072 15302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:10:53.553776 15302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:33031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:53.555106 15302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:53.556416 15302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:53.569674 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:53.569646 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:53.569723 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:53.570561 15302 server_base.cc:1034] running on GCE node
I20260620 20:10:53.571226 15302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:53.572265 15302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:53.573513 15302 hybrid_clock.cc:648] HybridClock initialized: now 1781986253573457 us; error 61 us; skew 500 ppm
I20260620 20:10:53.573890 15302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:53 dist-test-slave-4bwq krb5kdc[15062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986253, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:53.577210 15302 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:10:53.578804 15302 webserver.cc:460] Webserver started at http://127.4.169.130:46063/ using document root <none> and password file <none>
I20260620 20:10:53.579514 15302 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:53.579656 15302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:53.580173 15302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:53.583549 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "6bf165fbff1e4f03836d0069edc55462"
format_stamp: "Formatted at 2026-06-20 20:10:53 on dist-test-slave-4bwq"
server_key: "e2086f046cc0195a9b81dccbb3d944b1"
server_key_iv: "809e403a1f829c182ce37e008cbfe62c"
server_key_version: "encryptionkey@0"
I20260620 20:10:53.584381 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "6bf165fbff1e4f03836d0069edc55462"
format_stamp: "Formatted at 2026-06-20 20:10:53 on dist-test-slave-4bwq"
server_key: "e2086f046cc0195a9b81dccbb3d944b1"
server_key_iv: "809e403a1f829c182ce37e008cbfe62c"
server_key_version: "encryptionkey@0"
I20260620 20:10:53.589792 15302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:10:53.593369 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:53.594774 15302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:53.594944 15302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "6bf165fbff1e4f03836d0069edc55462"
format_stamp: "Formatted at 2026-06-20 20:10:53 on dist-test-slave-4bwq"
server_key: "e2086f046cc0195a9b81dccbb3d944b1"
server_key_iv: "809e403a1f829c182ce37e008cbfe62c"
server_key_version: "encryptionkey@0"
I20260620 20:10:53.595112 15302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:53.625775 15302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:53.629755 15302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:53.630038 15302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:53.631048 15302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:53.632371 15302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:53.632447 15302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:53.632573 15302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:53.632630 15302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:53.666692 15302 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:36963
I20260620 20:10:53.666740 15429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:36963 every 8 connection(s)
I20260620 20:10:53.668008 15302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:10:53.670809 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 15302
I20260620 20:10:53.671000 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:10:53.671558 4774 external_mini_cluster.cc:1428] Setting key c822452e46ea3370b1abf6e199f36e9b
Jun 20 20:10:53 dist-test-slave-4bwq krb5kdc[15062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986253, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:53.692271 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:53.670737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:37943 (local address 127.4.169.190:33031)
0620 20:10:53.671229 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:53.671236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:53.672589 (+ 1353us) server_negotiation.cc:407] Connection header received
0620 20:10:53.675111 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:53.675115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:53.675210 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:53.675298 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:53.677405 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:53.678589 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:53.681119 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:53.681559 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:53.681663 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:53.685417 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:53.685426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:53.685434 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:53.685480 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:53.688963 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:53.689653 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:53.689661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:53.689667 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:53.689822 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:53.690250 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:53.690257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:53.690263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:53.690677 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:53.690834 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:53.691505 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 20:10:53.691750 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260620 20:10:53.692876 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:53.671541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:33031 (local address 127.4.169.130:37943)
0620 20:10:53.672420 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:53.672469 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:53.674828 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:53.675588 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:53.675603 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:53.676391 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:53.677211 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:53.677239 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:53.678742 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:53.678750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:53.680936 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:53.680955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:53.681803 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:53.681813 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:53.682092 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:53.682831 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:53.682854 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:53.685172 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:53.689091 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:53.689098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:53.689111 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:53.689530 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:53.689930 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:53.689933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:53.689935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:53.690115 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:53.690798 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:53.690804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:53.691123 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 20:10:53.691378 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 20:10:53.691751 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":367,"threads_started":1}
I20260620 20:10:53.694445 15430 heartbeater.cc:347] Connected to a master server at 127.4.169.190:33031
I20260620 20:10:53.694802 15430 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:53.695766 15430 heartbeater.cc:511] Master 127.4.169.190:33031 requested a full tablet report, sending...
I20260620 20:10:53.697429 15107 ts_manager.cc:194] Registered new tserver with Master: 6bf165fbff1e4f03836d0069edc55462 (127.4.169.130:36963)
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:10:53.698365 15107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:37943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:10:53.725674 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:33031
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:10:54.057186 15436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:10:54.057463 15436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:10:54.057557 15436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:10:54.064927 15436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:10:54.065042 15436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:10:54.065222 15436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:10:54.075572 15436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:33031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:10:54.077046 15436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:10:54.078349 15436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:10:54.091614 15444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:54.091696 15442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:10:54.091717 15441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:10:54.092571 15436 server_base.cc:1034] running on GCE node
I20260620 20:10:54.093226 15436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:10:54.094152 15436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:10:54.095389 15436 hybrid_clock.cc:648] HybridClock initialized: now 1781986254095331 us; error 54 us; skew 500 ppm
I20260620 20:10:54.095722 15436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:10:54 dist-test-slave-4bwq krb5kdc[15062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986254, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:10:54.099041 15436 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:10:54.100521 15436 webserver.cc:460] Webserver started at http://127.4.169.131:46213/ using document root <none> and password file <none>
I20260620 20:10:54.101476 15436 fs_manager.cc:359] Metadata directory not provided
I20260620 20:10:54.101653 15436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:10:54.102008 15436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:10:54.104956 15436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "6fd45756d19a4066aaf08322fd95086e"
format_stamp: "Formatted at 2026-06-20 20:10:54 on dist-test-slave-4bwq"
server_key: "a5becc76e0e7ab7f7b8d13ec37741d43"
server_key_iv: "55724565bd6f6e25e593174bd539bbf7"
server_key_version: "encryptionkey@0"
I20260620 20:10:54.105739 15436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "6fd45756d19a4066aaf08322fd95086e"
format_stamp: "Formatted at 2026-06-20 20:10:54 on dist-test-slave-4bwq"
server_key: "a5becc76e0e7ab7f7b8d13ec37741d43"
server_key_iv: "55724565bd6f6e25e593174bd539bbf7"
server_key_version: "encryptionkey@0"
I20260620 20:10:54.110791 15436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:10:54.114002 15451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:54.115334 15436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:10:54.115495 15436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "6fd45756d19a4066aaf08322fd95086e"
format_stamp: "Formatted at 2026-06-20 20:10:54 on dist-test-slave-4bwq"
server_key: "a5becc76e0e7ab7f7b8d13ec37741d43"
server_key_iv: "55724565bd6f6e25e593174bd539bbf7"
server_key_version: "encryptionkey@0"
I20260620 20:10:54.115693 15436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:10:54.143497 15436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:10:54.147989 15436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:10:54.148288 15436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:10:54.149277 15436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:10:54.150704 15436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:10:54.150802 15436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:54.150944 15436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:10:54.151018 15436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:10:54.181131 15296 heartbeater.cc:503] Master 127.4.169.190:33031 was elected leader, sending a full tablet report...
I20260620 20:10:54.185868 15436 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:40185
I20260620 20:10:54.185917 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:40185 every 8 connection(s)
I20260620 20:10:54.187240 15436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:10:54.188609 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 15436
I20260620 20:10:54.188784 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:10:54.189308 4774 external_mini_cluster.cc:1428] Setting key 8f94e65ccacd815551a739c61d5e3769
Jun 20 20:10:54 dist-test-slave-4bwq krb5kdc[15062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986254, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:10:54.207756 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.189999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:51085 (local address 127.4.169.190:33031)
0620 20:10:54.190235 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.190243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.191634 (+ 1391us) server_negotiation.cc:407] Connection header received
0620 20:10:54.193573 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.193578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.193673 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.193789 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:54.195345 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.196061 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.197581 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.197973 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.198104 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.201559 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:54.201564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:54.201567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:54.201599 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:54.204741 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.205455 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.205460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.205464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.205580 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.206013 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.206018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.206021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.206293 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:54.206472 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.207205 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.207418 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 20:10:54.208695 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.190656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:33031 (local address 127.4.169.131:51085)
0620 20:10:54.191456 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:54.191509 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:54.193316 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:54.194028 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:54.194037 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:54.194619 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:10:54.195190 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.195206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.196212 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.196217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.197429 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.197440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.198188 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.198195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.198492 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.199190 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:10:54.199211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:10:54.201360 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:10:54.204881 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:54.204896 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:54.204909 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:54.205331 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:54.205707 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:10:54.205714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:10:54.205722 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:10:54.205898 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:10:54.206456 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:10:54.206464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:10:54.206823 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 20:10:54.207097 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 20:10:54.207638 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":290,"threads_started":1}
I20260620 20:10:54.210114 15564 heartbeater.cc:347] Connected to a master server at 127.4.169.190:33031
I20260620 20:10:54.210429 15564 heartbeater.cc:464] Registering TS with master...
I20260620 20:10:54.211131 15564 heartbeater.cc:511] Master 127.4.169.190:33031 requested a full tablet report, sending...
I20260620 20:10:54.212458 15107 ts_manager.cc:194] Registered new tserver with Master: 6fd45756d19a4066aaf08322fd95086e (127.4.169.131:40185)
I20260620 20:10:54.213201 15107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:51085
I20260620 20:10:54.216825 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:10:54.233052 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.221306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.4.169.190:33031)
0620 20:10:54.221518 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.221528 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.221896 (+ 368us) server_negotiation.cc:407] Connection header received
0620 20:10:54.222033 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.222036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.222092 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.222175 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:54.223560 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.224271 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.225416 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.225801 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.225888 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.228036 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:54.228041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:54.228044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:54.228074 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:54.230649 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.231283 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.231287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.231289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.231405 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.231753 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.231756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.231758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.232041 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:54.232227 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.232521 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.232671 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:10:54.237154 15107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:10:54.243886 15107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:10:54.266489 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.257552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.4.169.130:36963)
0620 20:10:54.259077 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.259087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.259119 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:10:54.259230 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.259240 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.259472 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.259678 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:54.260331 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.261893 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.263910 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.265281 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.265426 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.265612 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.265939 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.266111 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":1229,"threads_started":1}
I20260620 20:10:54.266566 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.257194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:36963 (local address 127.0.0.1:53828)
0620 20:10:54.258357 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:54.258380 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:54.258501 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:54.259667 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:54.259674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:54.259701 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:54.260138 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.260156 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.262025 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.262030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.263752 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.263764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.265472 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.265478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.265677 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.265727 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 20:10:54.265876 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:10:54.266071 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":2,"thread_start_us":509,"threads_started":1}
I20260620 20:10:54.267103 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.257168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:40185 (local address 127.0.0.1:34384)
0620 20:10:54.257732 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:54.257750 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:54.257856 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:54.259994 (+ 2138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:54.259999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:54.260029 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:54.260478 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.260489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.262277 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.262282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.264531 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.264552 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.266257 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.266264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.266484 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.266529 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 20:10:54.266626 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 20:10:54.266756 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":927,"threads_started":1}
I20260620 20:10:54.267529 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.256112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:41765 (local address 127.0.0.1:37608)
0620 20:10:54.256795 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:54.256830 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:54.257015 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:54.259169 (+ 2154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:54.259176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:54.259225 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:54.259556 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.259565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.261725 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.261731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.264298 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.264313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.266624 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.266631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.266828 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.266865 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:10:54.266938 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:10:54.267079 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":1,"thread_start_us":294,"threads_started":1}
I20260620 20:10:54.267992 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.256179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.4.169.129:41765)
0620 20:10:54.256772 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.256781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.258373 (+ 1592us) server_negotiation.cc:407] Connection header received
0620 20:10:54.258545 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.258556 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.258890 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.259759 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:54.260030 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.261558 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.264959 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.266332 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.267195 (+ 863us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.267418 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.267487 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.267639 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":297,"threads_started":1}
I20260620 20:10:54.268217 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.257518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.4.169.131:40185)
0620 20:10:54.259425 (+ 1907us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.259433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.259464 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:10:54.259562 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.259572 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.259798 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.260001 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:54.260684 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.262126 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.264722 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.266038 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.267471 (+ 1433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.267724 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.267800 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.267921 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1760,"thread_start_us":1595,"threads_started":1}
I20260620 20:10:54.271101 15365 tablet_service.cc:1467] Processing CreateTablet for tablet 568dd21478bd4a5f9f3e43e66447625d (DEFAULT_TABLE table=test-table [id=94238cd8ffd4422ab89677c93c877822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:54.271394 15231 tablet_service.cc:1467] Processing CreateTablet for tablet 568dd21478bd4a5f9f3e43e66447625d (DEFAULT_TABLE table=test-table [id=94238cd8ffd4422ab89677c93c877822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:54.271364 15499 tablet_service.cc:1467] Processing CreateTablet for tablet 568dd21478bd4a5f9f3e43e66447625d (DEFAULT_TABLE table=test-table [id=94238cd8ffd4422ab89677c93c877822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:10:54.272569 15231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568dd21478bd4a5f9f3e43e66447625d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:54.272595 15365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568dd21478bd4a5f9f3e43e66447625d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:54.272603 15499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568dd21478bd4a5f9f3e43e66447625d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:10:54.284765 15582 tablet_bootstrap.cc:492] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: Bootstrap starting.
I20260620 20:10:54.287777 15584 tablet_bootstrap.cc:492] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: Bootstrap starting.
I20260620 20:10:54.288887 15583 tablet_bootstrap.cc:492] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: Bootstrap starting.
I20260620 20:10:54.290261 15582 tablet_bootstrap.cc:654] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:54.291667 15582 log.cc:824] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:54.291749 15584 tablet_bootstrap.cc:654] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:54.292538 15583 tablet_bootstrap.cc:654] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: Neither blocks nor log segments found. Creating new log.
I20260620 20:10:54.292928 15584 log.cc:824] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:54.293608 15583 log.cc:824] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: Log is configured to *not* fsync() on all Append() calls
I20260620 20:10:54.294960 15582 tablet_bootstrap.cc:492] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: No bootstrap required, opened a new log
I20260620 20:10:54.295231 15582 ts_tablet_manager.cc:1397] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260620 20:10:54.295936 15584 tablet_bootstrap.cc:492] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: No bootstrap required, opened a new log
I20260620 20:10:54.296175 15584 ts_tablet_manager.cc:1397] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260620 20:10:54.299189 15583 tablet_bootstrap.cc:492] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: No bootstrap required, opened a new log
I20260620 20:10:54.299424 15583 ts_tablet_manager.cc:1397] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260620 20:10:54.306121 15583 raft_consensus.cc:348] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } }
I20260620 20:10:54.306119 15582 raft_consensus.cc:348] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } }
I20260620 20:10:54.306569 15582 raft_consensus.cc:374] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:54.306579 15583 raft_consensus.cc:374] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:54.306651 15582 raft_consensus.cc:729] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd45756d19a4066aaf08322fd95086e, State: Initialized, Role: FOLLOWER
I20260620 20:10:54.306670 15583 raft_consensus.cc:729] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e621f546f64108bea9a940eded17fc, State: Initialized, Role: FOLLOWER
I20260620 20:10:54.306490 15584 raft_consensus.cc:348] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } }
I20260620 20:10:54.306870 15584 raft_consensus.cc:374] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:10:54.306958 15584 raft_consensus.cc:729] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf165fbff1e4f03836d0069edc55462, State: Initialized, Role: FOLLOWER
I20260620 20:10:54.307337 15582 consensus_queue.cc:260] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } }
I20260620 20:10:54.307497 15584 consensus_queue.cc:260] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } }
I20260620 20:10:54.307494 15583 consensus_queue.cc:260] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } }
I20260620 20:10:54.309056 15564 heartbeater.cc:503] Master 127.4.169.190:33031 was elected leader, sending a full tablet report...
I20260620 20:10:54.309930 15582 ts_tablet_manager.cc:1428] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 20:10:54.310912 15430 heartbeater.cc:503] Master 127.4.169.190:33031 was elected leader, sending a full tablet report...
I20260620 20:10:54.311569 15584 ts_tablet_manager.cc:1428] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: Time spent starting tablet: real 0.015s user 0.005s sys 0.010s
I20260620 20:10:54.314273 15583 ts_tablet_manager.cc:1428] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260620 20:10:54.321470 15590 raft_consensus.cc:482] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:10:54.321698 15590 raft_consensus.cc:504] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } }
I20260620 20:10:54.323280 15590 leader_election.cc:288] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bf165fbff1e4f03836d0069edc55462 (127.4.169.130:36963), 6fd45756d19a4066aaf08322fd95086e (127.4.169.131:40185)
I20260620 20:10:54.332417 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.324035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:51925 (local address 127.4.169.130:36963)
0620 20:10:54.324280 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.324285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.324669 (+ 384us) server_negotiation.cc:407] Connection header received
0620 20:10:54.324767 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.324772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.324843 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.324927 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:54.325708 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.326831 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.329861 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.331121 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.331272 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.331344 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.332019 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.332151 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:10:54.332417 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.323787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:34079 (local address 127.4.169.131:40185)
0620 20:10:54.324098 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.324107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.324386 (+ 279us) server_negotiation.cc:407] Connection header received
0620 20:10:54.324715 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.324722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.324800 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.324906 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:10:54.325709 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.326855 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.329862 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.331136 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.331274 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.331344 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.332019 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.332177 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260620 20:10:54.332639 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.323586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:40185 (local address 127.4.169.129:34079)
0620 20:10:54.324272 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:54.324296 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:54.324502 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:54.324945 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:54.324950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:54.325036 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:54.325522 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.325541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.326972 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.326976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.329690 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.329706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.331435 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.331440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.331683 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.331857 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 20:10:54.331956 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:10:54.332141 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":405,"spinlock_wait_cycles":21760,"thread_start_us":684,"threads_started":1}
I20260620 20:10:54.332948 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.323703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:36963 (local address 127.4.169.129:51925)
0620 20:10:54.324554 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 20:10:54.324572 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:10:54.324683 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:10:54.324945 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:10:54.324950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:10:54.324974 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:10:54.325561 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.325570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.326971 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.326976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.329690 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:10:54.329706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.331416 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.331424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:10:54.331682 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.331857 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 20:10:54.331956 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:10:54.332141 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":283,"thread_start_us":553,"threads_started":1}
I20260620 20:10:54.333702 15519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568dd21478bd4a5f9f3e43e66447625d" candidate_uuid: "e3e621f546f64108bea9a940eded17fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd45756d19a4066aaf08322fd95086e" is_pre_election: true
I20260620 20:10:54.334033 15385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568dd21478bd4a5f9f3e43e66447625d" candidate_uuid: "e3e621f546f64108bea9a940eded17fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf165fbff1e4f03836d0069edc55462" is_pre_election: true
I20260620 20:10:54.334302 15519 raft_consensus.cc:2454] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3e621f546f64108bea9a940eded17fc in term 0.
I20260620 20:10:54.334522 15385 raft_consensus.cc:2454] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3e621f546f64108bea9a940eded17fc in term 0.
I20260620 20:10:54.335041 15184 leader_election.cc:302] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fd45756d19a4066aaf08322fd95086e, e3e621f546f64108bea9a940eded17fc; no voters:
I20260620 20:10:54.335390 15590 raft_consensus.cc:2781] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:10:54.335547 15590 raft_consensus.cc:482] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:10:54.335659 15590 raft_consensus.cc:3037] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:54.337944 15590 raft_consensus.cc:504] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } }
I20260620 20:10:54.338647 15590 leader_election.cc:288] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [CANDIDATE]: Term 1 election: Requested vote from peers 6bf165fbff1e4f03836d0069edc55462 (127.4.169.130:36963), 6fd45756d19a4066aaf08322fd95086e (127.4.169.131:40185)
I20260620 20:10:54.338980 15385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568dd21478bd4a5f9f3e43e66447625d" candidate_uuid: "e3e621f546f64108bea9a940eded17fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf165fbff1e4f03836d0069edc55462"
I20260620 20:10:54.339049 15519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568dd21478bd4a5f9f3e43e66447625d" candidate_uuid: "e3e621f546f64108bea9a940eded17fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd45756d19a4066aaf08322fd95086e"
I20260620 20:10:54.339181 15385 raft_consensus.cc:3037] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:54.339231 15519 raft_consensus.cc:3037] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:10:54.342365 15519 raft_consensus.cc:2454] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3e621f546f64108bea9a940eded17fc in term 1.
I20260620 20:10:54.342365 15385 raft_consensus.cc:2454] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3e621f546f64108bea9a940eded17fc in term 1.
I20260620 20:10:54.342814 15185 leader_election.cc:302] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6bf165fbff1e4f03836d0069edc55462, 6fd45756d19a4066aaf08322fd95086e, e3e621f546f64108bea9a940eded17fc; no voters:
I20260620 20:10:54.343132 15590 raft_consensus.cc:2781] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:10:54.343564 15590 raft_consensus.cc:686] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 1 LEADER]: Becoming Leader. State: Replica: e3e621f546f64108bea9a940eded17fc, State: Running, Role: LEADER
I20260620 20:10:54.344002 15590 consensus_queue.cc:237] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } }
I20260620 20:10:54.348476 15105 catalog_manager.cc:5495] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc reported cstate change: term changed from 0 to 1, leader changed from <none> to e3e621f546f64108bea9a940eded17fc (127.4.169.129). New cstate: current_term: 1 leader_uuid: "e3e621f546f64108bea9a940eded17fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3e621f546f64108bea9a940eded17fc" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 } health_report { overall_health: UNKNOWN } } }
I20260620 20:10:54.367822 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.362747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.4.169.129:41765)
0620 20:10:54.362926 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.362930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.363001 (+ 71us) server_negotiation.cc:407] Connection header received
0620 20:10:54.363167 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.363171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.363249 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.363371 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:10:54.363686 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.364388 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.365850 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.366244 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.366335 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.366828 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:10:54.367342 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:10:54.367448 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.367541 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.367602 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:10:54.376194 15385 raft_consensus.cc:1261] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [term 1 FOLLOWER]: Refusing update from remote peer e3e621f546f64108bea9a940eded17fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:54.376196 15519 raft_consensus.cc:1261] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [term 1 FOLLOWER]: Refusing update from remote peer e3e621f546f64108bea9a940eded17fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:10:54.377231 15593 consensus_queue.cc:1035] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf165fbff1e4f03836d0069edc55462" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:54.377555 15590 consensus_queue.cc:1035] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fd45756d19a4066aaf08322fd95086e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:10:54.393296 15598 mvcc.cc:205] Tried to move back new op lower bound from 7299015697914785792 to 7299015697796603904. Current Snapshot: MvccSnapshot[applied={T|T < 7299015697914785792}]
I20260620 20:10:54.402772 15105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:54.403067 15105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:10:54.406745 15105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P daab4aab6d7142dc8912a602207a9cb8: Sending DeleteTablet for 3 replicas of tablet 568dd21478bd4a5f9f3e43e66447625d
I20260620 20:10:54.407795 15231 tablet_service.cc:1514] Processing DeleteTablet for tablet 568dd21478bd4a5f9f3e43e66447625d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:54 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:37608
I20260620 20:10:54.407934 15499 tablet_service.cc:1514] Processing DeleteTablet for tablet 568dd21478bd4a5f9f3e43e66447625d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:54 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:34384
I20260620 20:10:54.407934 15365 tablet_service.cc:1514] Processing DeleteTablet for tablet 568dd21478bd4a5f9f3e43e66447625d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:10:54 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:53828
I20260620 20:10:54.410511 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 15168
I20260620 20:10:54.420408 15605 tablet_replica.cc:330] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: stopping tablet replica
I20260620 20:10:54.420893 15605 raft_consensus.cc:2229] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:10:54.421716 15605 raft_consensus.cc:2258] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:54.426040 15605 ts_tablet_manager.cc:1905] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:54.435452 15605 ts_tablet_manager.cc:1918] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:54.435664 15605 log.cc:1196] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/568dd21478bd4a5f9f3e43e66447625d
I20260620 20:10:54.436295 15605 ts_tablet_manager.cc:1939] T 568dd21478bd4a5f9f3e43e66447625d P e3e621f546f64108bea9a940eded17fc: Deleting consensus metadata
I20260620 20:10:54.438314 15092 catalog_manager.cc:4841] TS e3e621f546f64108bea9a940eded17fc (127.4.169.129:41765): tablet 568dd21478bd4a5f9f3e43e66447625d (table test-table [id=94238cd8ffd4422ab89677c93c877822]) successfully deleted
I20260620 20:10:54.439170 15607 tablet_replica.cc:330] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: stopping tablet replica
I20260620 20:10:54.439460 15606 tablet_replica.cc:330] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: stopping tablet replica
I20260620 20:10:54.439626 15607 raft_consensus.cc:2229] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:54.440174 15607 raft_consensus.cc:2258] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:54.440243 15606 raft_consensus.cc:2229] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:10:54.440785 15606 raft_consensus.cc:2258] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:10:54.442623 15607 ts_tablet_manager.cc:1905] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:10:54.442822 15606 ts_tablet_manager.cc:1905] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 20:10:54 dist-test-slave-4bwq krb5kdc[15062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:10:54.449607 15607 ts_tablet_manager.cc:1918] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:54.449774 15607 log.cc:1196] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/568dd21478bd4a5f9f3e43e66447625d
I20260620 20:10:54.450187 15606 ts_tablet_manager.cc:1918] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:10:54.450316 15607 ts_tablet_manager.cc:1939] T 568dd21478bd4a5f9f3e43e66447625d P 6bf165fbff1e4f03836d0069edc55462: Deleting consensus metadata
I20260620 20:10:54.450484 15606 log.cc:1196] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/568dd21478bd4a5f9f3e43e66447625d
I20260620 20:10:54.451392 15606 ts_tablet_manager.cc:1939] T 568dd21478bd4a5f9f3e43e66447625d P 6fd45756d19a4066aaf08322fd95086e: Deleting consensus metadata
I20260620 20:10:54.452280 15092 catalog_manager.cc:4841] TS 6bf165fbff1e4f03836d0069edc55462 (127.4.169.130:36963): tablet 568dd21478bd4a5f9f3e43e66447625d (table test-table [id=94238cd8ffd4422ab89677c93c877822]) successfully deleted
I20260620 20:10:54.453493 15091 catalog_manager.cc:4841] TS 6fd45756d19a4066aaf08322fd95086e (127.4.169.131:40185): tablet 568dd21478bd4a5f9f3e43e66447625d (table test-table [id=94238cd8ffd4422ab89677c93c877822]) successfully deleted
I20260620 20:10:54.456555 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.412294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.4.169.129:41765)
0620 20:10:54.412521 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.412542 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.437838 (+ 25296us) server_negotiation.cc:407] Connection header received
0620 20:10:54.438523 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.438528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.438620 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.438702 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:54.440683 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.441836 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.443633 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.444307 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.444406 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.449647 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:54.449667 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:54.449686 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:54.449741 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:54.453592 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.454319 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.454326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.454328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.454555 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.454950 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.454955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.454959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.455386 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:54.455705 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.455960 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.456192 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:10:54.457229 15291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:54.565944 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 15168
I20260620 20:10:54.585083 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 15302
Jun 20 20:10:54 dist-test-slave-4bwq krb5kdc[15062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:10:54.600360 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.585857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.4.169.130:36963)
0620 20:10:54.586129 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.586140 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.586162 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:10:54.586348 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.586355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.586479 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.586586 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:54.587819 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.588558 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.589834 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.590241 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.590385 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.594511 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:54.594528 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:54.594544 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:54.594585 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:54.597313 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.598031 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.598038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.598041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.598171 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.598805 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.598812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.598815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.599191 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:54.599506 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.599806 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.600021 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 20:10:54.601007 15425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:54.689465 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 15302
I20260620 20:10:54.705983 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 15436
Jun 20 20:10:54 dist-test-slave-4bwq krb5kdc[15062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:10:54.721341 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:10:54.706766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.4.169.131:40185)
0620 20:10:54.706958 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:10:54.706966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:10:54.707002 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:10:54.707286 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:10:54.707292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:10:54.707365 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:10:54.707470 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:10:54.708683 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.709427 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.710827 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:10:54.711228 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:10:54.711359 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:10:54.715723 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:10:54.715738 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:10:54.715754 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:10:54.715794 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:10:54.718457 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.719205 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.719217 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.719220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.719364 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:10:54.719756 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:10:54.719762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:10:54.719765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:10:54.720196 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:10:54.720485 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 20:10:54.720796 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 20:10:54.721014 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:10:54.721956 15559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:54.813939 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 15436
I20260620 20:10:54.831097 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 15075
I20260620 20:10:54.832213 15137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:10:54.938043 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 15075
2026-06-20T20:10:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4164 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:10:55 dist-test-slave-4bwq krb5kdc[15615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:10:55 dist-test-slave-4bwq krb5kdc[15615](info): set up 2 sockets
Jun 20 20:10:55 dist-test-slave-4bwq krb5kdc[15615](info): commencing operation
krb5kdc: starting...
W20260620 20:10:56.035558 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:10:56 dist-test-slave-4bwq krb5kdc[15615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:10:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:10:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.169.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.169.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/rangeradmin_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/rangeradmin_127.4.169.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.169.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.169.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/rangerlookup_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/rangerlookup_127.4.169.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.169.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.169.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/HTTP_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/HTTP_127.4.169.148@KRBTEST.COM.keytab.
I20260620 20:10:56.349941 4774 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBzMLvU/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/postgres -l logfile start
2026-06-20 20:10:58.778 UTC [15645] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:10:58.779 UTC [15645] LOG: listening on IPv4 address "127.4.169.148", port 48789
2026-06-20 20:10:58.782 UTC [15645] LOG: listening on Unix socket "/tmp/.s.PGSQL.48789"
2026-06-20 20:10:58.803 UTC [15648] LOG: database system was shut down at 2026-06-20 20:10:57 UTC
2026-06-20 20:10:58.807 UTC [15645] LOG: database system is ready to accept connections
I20260620 20:10:59.787246 4774 mini_postgres.cc:96] Postgres bound to 48789
2026-06-20 20:10:59.805 UTC [15656] FATAL: database "slave" does not exist
127.4.169.148:48789 - accepting connections
I20260620 20:10:59.806123 4774 mini_ranger.cc:161] Starting Ranger...
I20260620 20:10:59.836158 4774 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:11:00.331743 4774 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:11:00.331909 4774 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-admin
2026-06-20 20:11:00,633 [I] DB FLAVOR :POSTGRES
2026-06-20 20:11:00,634 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:11:00,634 [I] Checking connection..
2026-06-20 20:11:00,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:11:00,885 [I] Checking connection passed.
2026-06-20 20:11:00,885 [I] --------- Verifying version history table ---------
2026-06-20 20:11:00,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:11:01,148 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:11:01,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:11:01,397 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:11:01,397 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:11:01,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:11:01.659 UTC [15720] WARNING: there is no transaction in progress
2026-06-20 20:11:01,665 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:11:01,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:11:01,931 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:11:01,931 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:11:01,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:11:02,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:11:02,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4bwq.c.gcp-upstream.internal','N') ;"
2026-06-20 20:11:02,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:11:02,910 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:11:02,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:11:03,160 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:11:03,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:11:03,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:11:03,668 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:11:03,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:11:04.639 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.654 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.670 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.685 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.700 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.718 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.780 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.787 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.801 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.811 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.821 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.835 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.844 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.851 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.857 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:04.863 UTC [15828] WARNING: there is no transaction in progress
2026-06-20 20:11:05,464 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:11:05,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:11:05,720 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:11:05,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:11:05,971 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:11:05,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:11:06,205 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:11:06,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:11:06,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:11:06,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4bwq.c.gcp-upstream.internal';"
2026-06-20 20:11:06,938 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:11:06,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:11:07,199 [I] DB_PATCHES have already been applied
I20260620 20:11:07.205638 4774 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBzMLvU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:11:07.205752 4774 mini_ranger.cc:193] Using host: 127.4.169.148
I20260620 20:11:07.221066 4774 mini_ranger.cc:227] Ranger admin URL: http://127.4.169.148:51663
Jun 20, 2026 8:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:11:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:11:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:11:08.363019 4774 mini_ranger.cc:160] Time spent starting Ranger: real 8.557s user 0.000s sys 0.042s
Jun 20, 2026 8:11:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:11:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 8:11:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:11:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:11:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:11:30.767807 4774 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.169.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.169.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/rangerkms_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/rangerkms_127.4.169.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.169.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.169.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.169.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/HTTP_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.169.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/HTTP_127.4.169.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:11:30 dist-test-slave-4bwq krb5kdc[15615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986290, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:11:30.931723 4774 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:11:30.954977 4774 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:11:31.783193 4774 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:11:31.783327 4774 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-kms
I20260620 20:11:31.783483 4774 mini_ranger_kms.cc:227] Using postgres at 127.4.169.148:48789
2026-06-20 20:11:32,213 [I] DB FLAVOR :POSTGRES
2026-06-20 20:11:32,214 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:11:32,214 [I] Checking connection
2026-06-20 20:11:32,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:11:32,456 [I] connection success
2026-06-20 20:11:32,456 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:11:32,456 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:11:32,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:11:32,710 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:11:32,710 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:11:32,710 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:11:32,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:48789/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:11:32,994 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:11:33.506542 4774 mini_ranger_kms.cc:310] Created kms service
I20260620 20:11:33.689106 4774 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:11:33.733315 4774 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:11:33 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986290, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:11:34.161772 4774 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:11:34.162096 4774 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:11:34.162168 4774 mini_ranger_kms.cc:240] Using host: 127.4.169.148
I20260620 20:11:34.180303 4774 mini_ranger_kms.cc:276] Ranger KMS PID: 16009
I20260620 20:11:34.180501 4774 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.169.148:49841
Jun 20, 2026 8:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 20:11:36.591675 4774 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.660s user 0.109s sys 0.344s
I20260620 20:11:36.591946 4774 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:11:36.592200 4774 mini_ranger_kms.cc:401] 127.4.169.148:49841/kms/v1/keys
Jun 20 20:11:36 dist-test-slave-4bwq krb5kdc[15615](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.169.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 20:11:37 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.169.148@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
Jun 20 20:11:37 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.169.148@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
Jun 20 20:11:39 dist-test-slave-4bwq krb5kdc[15615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:11:40.048128 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34761
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34761
--ranger_config_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:49841/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:11:40.422781 16059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:11:40.423168 16059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:11:40.423329 16059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:11:40.431010 16059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:11:40.431183 16059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:11:40.431294 16059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:11:40.431387 16059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:11:40.431473 16059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:11:40.442418 16059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.169.148:49841/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34761
--ranger_config_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:11:40.443882 16059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:11:40.445340 16059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:11:40.458642 16067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:11:40.458714 16064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:11:40.458688 16065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:11:40.459444 16059 server_base.cc:1034] running on GCE node
I20260620 20:11:40.460718 16059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:11:40.462366 16059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:11:40.463689 16059 hybrid_clock.cc:648] HybridClock initialized: now 1781986300463635 us; error 55 us; skew 500 ppm
I20260620 20:11:40.464111 16059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:11:40 dist-test-slave-4bwq krb5kdc[15615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986300, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:11:40.472193 16059 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:11:40.473869 16059 webserver.cc:460] Webserver started at http://127.4.169.190:45457/ using document root <none> and password file <none>
I20260620 20:11:40.474658 16059 fs_manager.cc:359] Metadata directory not provided
I20260620 20:11:40.474757 16059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:11:40.475026 16059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:11:40 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986300, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:11:40.539070 16059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "d287f4aa7bb948679d9824fd0d13c31a"
format_stamp: "Formatted at 2026-06-20 20:11:40 on dist-test-slave-4bwq"
server_key: "859d99f95cb7a9256b505991e2712d83"
server_key_iv: "b2dfd8b99b7d9358e1af2987bf5c424e"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:40.539916 16059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "d287f4aa7bb948679d9824fd0d13c31a"
format_stamp: "Formatted at 2026-06-20 20:11:40 on dist-test-slave-4bwq"
server_key: "859d99f95cb7a9256b505991e2712d83"
server_key_iv: "b2dfd8b99b7d9358e1af2987bf5c424e"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:40.545661 16059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:11:40.563573 16074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:11:40.565295 16059 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 20:11:40.565459 16059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "d287f4aa7bb948679d9824fd0d13c31a"
format_stamp: "Formatted at 2026-06-20 20:11:40 on dist-test-slave-4bwq"
server_key: "859d99f95cb7a9256b505991e2712d83"
server_key_iv: "b2dfd8b99b7d9358e1af2987bf5c424e"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:40.565672 16059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:11:40.588497 16059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:11:40.592753 16059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:11:40.593030 16059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:11:40.614118 16059 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:34761
I20260620 20:11:40.614128 16125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:34761 every 8 connection(s)
I20260620 20:11:40.615608 16059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:11:40.619590 16126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:11:40.621497 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 16059
I20260620 20:11:40.621708 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:11:40.622551 4774 external_mini_cluster.cc:1428] Setting key afb7b3d3769d830f417a73bbc85b07a9
I20260620 20:11:40.632791 16126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a: Bootstrap starting.
I20260620 20:11:40.636270 16126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a: Neither blocks nor log segments found. Creating new log.
Jun 20 20:11:40 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:11:40.637378 16126 log.cc:824] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:11:40.641623 16126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a: No bootstrap required, opened a new log
I20260620 20:11:40.647996 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:40.625314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.4.169.190:34761)
0620 20:11:40.626267 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:40.626284 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:40.626329 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:11:40.628193 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:40.628220 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:40.628647 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:40.629099 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:11:40.630547 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:40.631931 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:40.633346 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:40.634085 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:40.634305 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:40.638449 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:11:40.638465 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:11:40.638481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:11:40.638526 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:11:40.642914 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:40.643795 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:40.643808 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:40.643817 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:40.644012 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:40.644422 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:40.644427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:40.644431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:40.645225 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:11:40.645529 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:40.645973 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 20:11:40.646452 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":444,"threads_started":1}
I20260620 20:11:40.651984 16126 raft_consensus.cc:348] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287f4aa7bb948679d9824fd0d13c31a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34761 } }
I20260620 20:11:40.652351 16126 raft_consensus.cc:374] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:11:40.652426 16126 raft_consensus.cc:729] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d287f4aa7bb948679d9824fd0d13c31a, State: Initialized, Role: FOLLOWER
I20260620 20:11:40.652930 16126 consensus_queue.cc:260] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287f4aa7bb948679d9824fd0d13c31a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34761 } }
I20260620 20:11:40.653137 16126 raft_consensus.cc:388] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:11:40.653236 16126 raft_consensus.cc:482] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:11:40.653359 16126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:11:40.655562 16126 raft_consensus.cc:504] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287f4aa7bb948679d9824fd0d13c31a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34761 } }
I20260620 20:11:40.656054 16126 leader_election.cc:302] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d287f4aa7bb948679d9824fd0d13c31a; no voters:
I20260620 20:11:40.657001 16126 leader_election.cc:288] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:11:40.657158 16131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:11:40.658528 16131 raft_consensus.cc:686] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [term 1 LEADER]: Becoming Leader. State: Replica: d287f4aa7bb948679d9824fd0d13c31a, State: Running, Role: LEADER
I20260620 20:11:40.659137 16131 consensus_queue.cc:237] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287f4aa7bb948679d9824fd0d13c31a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34761 } }
I20260620 20:11:40.659403 16126 sys_catalog.cc:564] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:11:40.661931 16133 sys_catalog.cc:455] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d287f4aa7bb948679d9824fd0d13c31a. Latest consensus state: current_term: 1 leader_uuid: "d287f4aa7bb948679d9824fd0d13c31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287f4aa7bb948679d9824fd0d13c31a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34761 } } }
I20260620 20:11:40.663059 16133 sys_catalog.cc:458] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [sys.catalog]: This master's current role is: LEADER
I20260620 20:11:40.662536 16132 sys_catalog.cc:455] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d287f4aa7bb948679d9824fd0d13c31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d287f4aa7bb948679d9824fd0d13c31a" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34761 } } }
I20260620 20:11:40.663589 16132 sys_catalog.cc:458] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a [sys.catalog]: This master's current role is: LEADER
I20260620 20:11:40.667169 16140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:11:40.671995 16140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:11:40.672852 16126 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:11:40.690302 16140 catalog_manager.cc:1347] Generated new cluster ID: 7c85da61428440d19b5b291263334bae
I20260620 20:11:40.690555 16140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:11:40.711645 16140 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:11:40.713145 16140 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:11:40.727206 16140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a: Generated new TSK 0
I20260620 20:11:40.728400 16140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:11:41.558794 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:49841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34761
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:11:41 dist-test-slave-4bwq krb5kdc[15615](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986301, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 20:11:42.041005 16171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:11:42.041503 16171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:11:42.041690 16171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:11:42.053776 16171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:11:42.054014 16171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:11:42.054240 16171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:11:42.071053 16171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.169.148:49841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:11:42.073227 16171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:11:42.075178 16171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:11:42.095103 16182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:11:42.095103 16184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:11:42.096411 16171 server_base.cc:1034] running on GCE node
W20260620 20:11:42.098714 16181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:11:42.099553 16171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:11:42.100957 16171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:11:42.102219 16171 hybrid_clock.cc:648] HybridClock initialized: now 1781986302102163 us; error 56 us; skew 500 ppm
I20260620 20:11:42.102635 16171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:11:42 dist-test-slave-4bwq krb5kdc[15615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986302, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:11:42.106139 16171 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:11:42.112068 16171 webserver.cc:460] Webserver started at http://127.4.169.129:36955/ using document root <none> and password file <none>
I20260620 20:11:42.112859 16171 fs_manager.cc:359] Metadata directory not provided
I20260620 20:11:42.113016 16171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:11:42.113359 16171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:11:42 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986302, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:11:42.131515 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "d76a136637634f2f8b5d0f807627236c"
format_stamp: "Formatted at 2026-06-20 20:11:42 on dist-test-slave-4bwq"
server_key: "1921d612199863d49397418cb0344530"
server_key_iv: "c422954bb0350ed720d66ad7290fa043"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:42.133289 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "d76a136637634f2f8b5d0f807627236c"
format_stamp: "Formatted at 2026-06-20 20:11:42 on dist-test-slave-4bwq"
server_key: "1921d612199863d49397418cb0344530"
server_key_iv: "c422954bb0350ed720d66ad7290fa043"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:42.144829 16171 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.010s sys 0.000s
I20260620 20:11:42.179090 16191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:11:42.181368 16171 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.001s sys 0.014s
I20260620 20:11:42.181663 16171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "d76a136637634f2f8b5d0f807627236c"
format_stamp: "Formatted at 2026-06-20 20:11:42 on dist-test-slave-4bwq"
server_key: "1921d612199863d49397418cb0344530"
server_key_iv: "c422954bb0350ed720d66ad7290fa043"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:42.182006 16171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:11:42.248411 16171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:11:42.253938 16171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:11:42.254350 16171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:11:42.262459 16171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:11:42.271029 16171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:11:42.271236 16171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:11:42.271456 16171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:11:42.271643 16171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:11:42 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986301, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:11:42.538370 16171 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:39955
I20260620 20:11:42.538436 16303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:39955 every 8 connection(s)
I20260620 20:11:42.540386 16171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:11:42.543154 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 16171
I20260620 20:11:42.543367 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:11:42.544178 4774 external_mini_cluster.cc:1428] Setting key 330bfc3833b249feb9bd6ba69a1e6f1a
Jun 20 20:11:42 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986302, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:11:42.620067 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:42.544705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:37875 (local address 127.4.169.190:34761)
0620 20:11:42.565044 (+ 20339us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:42.565051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:42.568847 (+ 3796us) server_negotiation.cc:407] Connection header received
0620 20:11:42.587309 (+ 18462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:42.587317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:42.587423 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:42.587509 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:11:42.590800 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:42.591954 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:42.606028 (+ 14074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:42.606780 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:42.606931 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:42.612137 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:11:42.612145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:11:42.612150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:11:42.612204 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:11:42.615988 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:42.617077 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:42.617084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:42.617088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:42.617264 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:42.617831 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:42.617836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:42.617840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:42.618233 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:11:42.618370 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:42.619418 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0620 20:11:42.619678 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20185,"thread_start_us":20360,"threads_started":1}
I20260620 20:11:42.622761 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:42.544053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34761 (local address 127.4.169.129:37875)
0620 20:11:42.567378 (+ 23325us) negotiation.cc:107] Waiting for socket to connect
0620 20:11:42.567453 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 20:11:42.583714 (+ 16261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:11:42.588158 (+ 4444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:11:42.588172 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:11:42.588978 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:11:42.589911 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:42.589938 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:42.592186 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:42.592191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:42.594689 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:42.594717 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:42.607388 (+ 12671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:42.607397 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:42.607760 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:42.608714 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:11:42.608746 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:11:42.611785 (+ 3039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:11:42.616233 (+ 4448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:11:42.616243 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:11:42.616264 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:11:42.616919 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:11:42.617421 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:11:42.617426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:11:42.617430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:11:42.617695 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:11:42.618483 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:11:42.618492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:11:42.618965 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 20:11:42.620721 (+ 1756us) client_negotiation.cc:240] Negotiation successful
0620 20:11:42.621327 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22946,"thread_start_us":39989,"threads_started":1}
I20260620 20:11:42.624843 16304 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34761
I20260620 20:11:42.625171 16304 heartbeater.cc:464] Registering TS with master...
I20260620 20:11:42.625996 16304 heartbeater.cc:511] Master 127.4.169.190:34761 requested a full tablet report, sending...
I20260620 20:11:42.629215 16091 ts_manager.cc:194] Registered new tserver with Master: d76a136637634f2f8b5d0f807627236c (127.4.169.129:39955)
I20260620 20:11:42.631400 16091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:37875
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
W20260620 20:11:42.680747 4774 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.169.130: real 0.113s user 0.001s sys 0.016s
I20260620 20:11:42.681507 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:49841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34761
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:11:42 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986301, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
W20260620 20:11:43.155026 16310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:11:43.155319 16310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:11:43.155470 16310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:11:43.163028 16310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:11:43.163141 16310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:11:43.163335 16310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:11:43.175539 16310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.169.148:49841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:11:43.176908 16310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:11:43.178232 16310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:11:43.192014 16324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:11:43.192075 16327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:11:43.192051 16325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:11:43.194177 16310 server_base.cc:1034] running on GCE node
I20260620 20:11:43.194864 16310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:11:43.195921 16310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:11:43.197135 16310 hybrid_clock.cc:648] HybridClock initialized: now 1781986303197089 us; error 43 us; skew 500 ppm
I20260620 20:11:43.197453 16310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:11:43 dist-test-slave-4bwq krb5kdc[15615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:11:43.200636 16310 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:11:43.202260 16310 webserver.cc:460] Webserver started at http://127.4.169.130:34261/ using document root <none> and password file <none>
I20260620 20:11:43.202999 16310 fs_manager.cc:359] Metadata directory not provided
I20260620 20:11:43.203141 16310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:11:43.203469 16310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:11:43 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:11:43.220685 16310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "8b5f6a92bd894795a7de387c69bcf511"
format_stamp: "Formatted at 2026-06-20 20:11:43 on dist-test-slave-4bwq"
server_key: "7b6367d5cbd4aa6f004fc5cc16e8020d"
server_key_iv: "ed749300c9a6745a0696aa39e9383424"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:43.221621 16310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "8b5f6a92bd894795a7de387c69bcf511"
format_stamp: "Formatted at 2026-06-20 20:11:43 on dist-test-slave-4bwq"
server_key: "7b6367d5cbd4aa6f004fc5cc16e8020d"
server_key_iv: "ed749300c9a6745a0696aa39e9383424"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:43.227531 16310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:11:43.246415 16334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:11:43.248502 16310 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:11:43.248728 16310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "8b5f6a92bd894795a7de387c69bcf511"
format_stamp: "Formatted at 2026-06-20 20:11:43 on dist-test-slave-4bwq"
server_key: "7b6367d5cbd4aa6f004fc5cc16e8020d"
server_key_iv: "ed749300c9a6745a0696aa39e9383424"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:43.249002 16310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:11:43.281908 16310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:11:43.286082 16310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:11:43.286465 16310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:11:43.287577 16310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:11:43.289219 16310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:11:43.289317 16310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:11:43.289482 16310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:11:43.289564 16310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:11:43.333877 16310 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:42945
I20260620 20:11:43.333892 16446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:42945 every 8 connection(s)
I20260620 20:11:43.335353 16310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:11:43.344376 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 16310
I20260620 20:11:43.344552 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:11:43.345067 4774 external_mini_cluster.cc:1428] Setting key 51494dffe1fe80452a65efe63cc22827
Jun 20 20:11:43 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:11:43.358966 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:43.338172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:35633 (local address 127.4.169.190:34761)
0620 20:11:43.338678 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:43.338683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:43.339821 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 20:11:43.341617 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:43.341623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:43.341736 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:43.341858 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:11:43.343416 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.344152 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.346052 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.346656 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.346788 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:43.351401 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:11:43.351408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:11:43.351413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:11:43.351457 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:11:43.355600 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:43.356479 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:43.356485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:43.356488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:43.356626 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:43.357125 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:43.357131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:43.357134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:43.357422 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:11:43.357597 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:43.358317 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 20:11:43.358580 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
I20260620 20:11:43.360024 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:43.338878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34761 (local address 127.4.169.130:35633)
0620 20:11:43.339663 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 20:11:43.339701 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:11:43.341331 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:11:43.342114 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:11:43.342123 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:11:43.342637 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:11:43.343235 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:43.343253 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.344333 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.344337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:43.345877 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:43.345905 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.346917 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.346926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:43.347255 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:43.348097 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:11:43.348136 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:11:43.351099 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:11:43.355785 (+ 4686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:11:43.355798 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:11:43.355817 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:11:43.356330 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:11:43.356748 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:11:43.356753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:11:43.356756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:11:43.356990 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:11:43.357566 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:11:43.357574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:11:43.357932 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 20:11:43.358206 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 20:11:43.358807 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":327,"threads_started":1}
I20260620 20:11:43.361450 16447 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34761
I20260620 20:11:43.361783 16447 heartbeater.cc:464] Registering TS with master...
I20260620 20:11:43.362599 16447 heartbeater.cc:511] Master 127.4.169.190:34761 requested a full tablet report, sending...
I20260620 20:11:43.364182 16091 ts_manager.cc:194] Registered new tserver with Master: 8b5f6a92bd894795a7de387c69bcf511 (127.4.169.130:42945)
I20260620 20:11:43.365056 16091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:35633
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:11:43.406519 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:49841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34761
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:11:43.635866 16304 heartbeater.cc:503] Master 127.4.169.190:34761 was elected leader, sending a full tablet report...
W20260620 20:11:43.772287 16453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:11:43.772665 16453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:11:43.772842 16453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:11:43.780761 16453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:11:43.780927 16453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:11:43.781136 16453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:11:43.791620 16453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.169.148:49841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:11:43.792969 16453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:11:43.794286 16453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:11:43.808640 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:11:43.808691 16461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:11:43.808748 16458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:11:43.809553 16453 server_base.cc:1034] running on GCE node
I20260620 20:11:43.810252 16453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:11:43.811309 16453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:11:43.813565 16453 hybrid_clock.cc:648] HybridClock initialized: now 1781986303813490 us; error 67 us; skew 500 ppm
I20260620 20:11:43.813943 16453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:11:43 dist-test-slave-4bwq krb5kdc[15615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:11:43.817720 16453 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:11:43.819634 16453 webserver.cc:460] Webserver started at http://127.4.169.131:46415/ using document root <none> and password file <none>
I20260620 20:11:43.820384 16453 fs_manager.cc:359] Metadata directory not provided
I20260620 20:11:43.820493 16453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:11:43.820775 16453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:11:43 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:11:43.835809 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "a43bfecb31e04671a26df497be91f31e"
format_stamp: "Formatted at 2026-06-20 20:11:43 on dist-test-slave-4bwq"
server_key: "08e2103388aec5ac5d705b8dfe31cb70"
server_key_iv: "28692fab4aaa7248b44ae2616ccfd272"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:43.836967 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "a43bfecb31e04671a26df497be91f31e"
format_stamp: "Formatted at 2026-06-20 20:11:43 on dist-test-slave-4bwq"
server_key: "08e2103388aec5ac5d705b8dfe31cb70"
server_key_iv: "28692fab4aaa7248b44ae2616ccfd272"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:43.844151 16453 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260620 20:11:43.860728 16468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:11:43.862495 16453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 20:11:43.862751 16453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "a43bfecb31e04671a26df497be91f31e"
format_stamp: "Formatted at 2026-06-20 20:11:43 on dist-test-slave-4bwq"
server_key: "08e2103388aec5ac5d705b8dfe31cb70"
server_key_iv: "28692fab4aaa7248b44ae2616ccfd272"
server_key_version: "kuduclusterkey@0"
I20260620 20:11:43.863054 16453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:11:43.904007 16453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:11:43.907985 16453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:11:43.908299 16453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:11:43.909274 16453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:11:43.910689 16453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:11:43.910791 16453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:11:43.910933 16453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:11:43.911005 16453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:11:43.949266 16453 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:38977
I20260620 20:11:43.949306 16580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:38977 every 8 connection(s)
I20260620 20:11:43.950664 16453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:11:43.954322 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 16453
I20260620 20:11:43.954543 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:11:43.955184 4774 external_mini_cluster.cc:1428] Setting key 22c83a19a284ef86775a71a7d41be15a
Jun 20 20:11:43 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:11:43.973426 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:43.953440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:45527 (local address 127.4.169.190:34761)
0620 20:11:43.954041 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:43.954049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:43.955044 (+ 995us) server_negotiation.cc:407] Connection header received
0620 20:11:43.957434 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:43.957438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:43.957521 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:43.957611 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:11:43.959417 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.960221 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.962055 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.962590 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.962711 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:43.966654 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:11:43.966660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:11:43.966663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:11:43.966702 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:11:43.970005 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:43.971084 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:43.971088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:43.971090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:43.971299 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:43.971818 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:43.971821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:43.971823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:43.972103 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:11:43.972221 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:43.972981 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 20:11:43.973162 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":274,"threads_started":1}
I20260620 20:11:43.974538 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:43.954028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34761 (local address 127.4.169.131:45527)
0620 20:11:43.954857 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 20:11:43.954904 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:11:43.957114 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:11:43.957922 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:11:43.957936 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:11:43.958587 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:11:43.959218 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:43.959237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.960430 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.960436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:43.961873 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:43.961886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.962940 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.962949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:43.963177 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:43.964059 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:11:43.964092 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:11:43.966303 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:11:43.970179 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:11:43.970187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:11:43.970203 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:11:43.970944 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:11:43.971430 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:11:43.971436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:11:43.971439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:11:43.971697 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:11:43.972235 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:11:43.972241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:11:43.972619 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 20:11:43.972893 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 20:11:43.973313 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":372,"threads_started":1}
I20260620 20:11:43.976081 16581 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34761
I20260620 20:11:43.976416 16581 heartbeater.cc:464] Registering TS with master...
I20260620 20:11:43.977267 16581 heartbeater.cc:511] Master 127.4.169.190:34761 requested a full tablet report, sending...
I20260620 20:11:43.978847 16091 ts_manager.cc:194] Registered new tserver with Master: a43bfecb31e04671a26df497be91f31e (127.4.169.131:38977)
I20260620 20:11:43.979728 16091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:45527
I20260620 20:11:43.983975 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:11:44.003383 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:43.988727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.4.169.190:34761)
0620 20:11:43.988896 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:43.988901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:43.989477 (+ 576us) server_negotiation.cc:407] Connection header received
0620 20:11:43.989667 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:43.989671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:43.989752 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:43.989842 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:11:43.991329 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.992061 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.993349 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:43.993762 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:43.993861 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:43.996211 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:11:43.996216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:11:43.996219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:11:43.996253 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:11:44.000219 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:44.001079 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:44.001089 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:44.001093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:44.001289 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:44.001823 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:44.001827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:44.001829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:44.002126 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:11:44.002303 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.002838 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.003097 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:11:44.007810 16091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:11:44.015194 16091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:11:44.053370 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.029216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.4.169.131:38977)
0620 20:11:44.042476 (+ 13260us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:44.042488 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:44.042534 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:11:44.042663 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:44.042672 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:44.042978 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:44.043193 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:11:44.044296 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.045954 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.049993 (+ 4039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.051496 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.051638 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.051894 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.052801 (+ 907us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.052990 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13067,"thread_start_us":12921,"threads_started":1}
I20260620 20:11:44.053687 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.028896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:38977 (local address 127.0.0.1:50928)
0620 20:11:44.042002 (+ 13106us) negotiation.cc:107] Waiting for socket to connect
0620 20:11:44.042066 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 20:11:44.042363 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:11:44.043494 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:11:44.043501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:11:44.043542 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:11:44.044058 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.044073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.046157 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.046165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.049772 (+ 3607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.049781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.052060 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.052077 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.052416 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.052549 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 20:11:44.052703 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 20:11:44.052969 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11676,"mutex_wait_us":440,"thread_start_us":13650,"threads_started":1}
I20260620 20:11:44.054261 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.029345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.4.169.130:42945)
0620 20:11:44.041691 (+ 12346us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:44.041705 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:44.043158 (+ 1453us) server_negotiation.cc:407] Connection header received
0620 20:11:44.043262 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:44.043269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:44.043505 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:44.043697 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:11:44.044658 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.046104 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.049889 (+ 3785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.051492 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.051638 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.051862 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.053712 (+ 1850us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.053913 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12117,"thread_start_us":12397,"threads_started":1}
I20260620 20:11:44.054561 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.029307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:42945 (local address 127.0.0.1:47932)
0620 20:11:44.042843 (+ 13536us) negotiation.cc:107] Waiting for socket to connect
0620 20:11:44.042862 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:11:44.043008 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:11:44.043955 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:11:44.043961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:11:44.043992 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:11:44.044444 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.044457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.046247 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.046254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.049706 (+ 3452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.049722 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.052093 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.052098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.052401 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.052550 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 20:11:44.052703 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 20:11:44.052980 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13409,"mutex_wait_us":1341,"thread_start_us":11613,"threads_started":1}
I20260620 20:11:44.057000 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.043059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39955 (local address 127.0.0.1:49130)
0620 20:11:44.047727 (+ 4668us) negotiation.cc:107] Waiting for socket to connect
0620 20:11:44.047748 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:11:44.047882 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:11:44.049218 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:11:44.049223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:11:44.049254 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:11:44.049715 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.049733 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.051484 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.051489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.053678 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.053696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.056092 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.056099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.056293 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.056337 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 20:11:44.056449 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:11:44.056607 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4512,"mutex_wait_us":1,"thread_start_us":4339,"threads_started":1}
I20260620 20:11:44.057044 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.043319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.4.169.129:39955)
0620 20:11:44.048499 (+ 5180us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:44.048509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:44.048540 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:11:44.048651 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:44.048660 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:44.048934 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:44.049165 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:11:44.049940 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.051303 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.053886 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.055552 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.055700 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.055932 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.056531 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.056711 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4988,"thread_start_us":4836,"threads_started":1}
I20260620 20:11:44.058557 16516 tablet_service.cc:1467] Processing CreateTablet for tablet 29a75aeeb7f7478ba998204cfa7aa363 (DEFAULT_TABLE table=test-table [id=ab8714ab3f8a49dfab7f6fe9ed10880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:11:44.060026 16516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a75aeeb7f7478ba998204cfa7aa363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:11:44.059573 16382 tablet_service.cc:1467] Processing CreateTablet for tablet 29a75aeeb7f7478ba998204cfa7aa363 (DEFAULT_TABLE table=test-table [id=ab8714ab3f8a49dfab7f6fe9ed10880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:11:44.061038 16382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a75aeeb7f7478ba998204cfa7aa363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:11:44.061208 16239 tablet_service.cc:1467] Processing CreateTablet for tablet 29a75aeeb7f7478ba998204cfa7aa363 (DEFAULT_TABLE table=test-table [id=ab8714ab3f8a49dfab7f6fe9ed10880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:11:44.062795 16239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a75aeeb7f7478ba998204cfa7aa363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:11:44.072850 16600 tablet_bootstrap.cc:492] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: Bootstrap starting.
I20260620 20:11:44.077147 16601 tablet_bootstrap.cc:492] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: Bootstrap starting.
I20260620 20:11:44.078116 16600 tablet_bootstrap.cc:654] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: Neither blocks nor log segments found. Creating new log.
I20260620 20:11:44.079394 16602 tablet_bootstrap.cc:492] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: Bootstrap starting.
I20260620 20:11:44.080008 16600 log.cc:824] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:11:44.082319 16601 tablet_bootstrap.cc:654] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: Neither blocks nor log segments found. Creating new log.
I20260620 20:11:44.083812 16601 log.cc:824] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: Log is configured to *not* fsync() on all Append() calls
I20260620 20:11:44.084003 16602 tablet_bootstrap.cc:654] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: Neither blocks nor log segments found. Creating new log.
I20260620 20:11:44.084100 16600 tablet_bootstrap.cc:492] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: No bootstrap required, opened a new log
I20260620 20:11:44.084424 16600 ts_tablet_manager.cc:1397] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 20:11:44.085891 16602 log.cc:824] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:11:44.087489 16601 tablet_bootstrap.cc:492] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: No bootstrap required, opened a new log
I20260620 20:11:44.087812 16601 ts_tablet_manager.cc:1397] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 20:11:44.089658 16602 tablet_bootstrap.cc:492] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: No bootstrap required, opened a new log
I20260620 20:11:44.089985 16602 ts_tablet_manager.cc:1397] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260620 20:11:44.096637 16601 raft_consensus.cc:348] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } }
I20260620 20:11:44.096707 16600 raft_consensus.cc:348] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } }
I20260620 20:11:44.097191 16600 raft_consensus.cc:374] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:11:44.097193 16601 raft_consensus.cc:374] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:11:44.097309 16601 raft_consensus.cc:729] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5f6a92bd894795a7de387c69bcf511, State: Initialized, Role: FOLLOWER
I20260620 20:11:44.097312 16600 raft_consensus.cc:729] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43bfecb31e04671a26df497be91f31e, State: Initialized, Role: FOLLOWER
I20260620 20:11:44.098069 16601 consensus_queue.cc:260] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } }
I20260620 20:11:44.098083 16600 consensus_queue.cc:260] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } }
I20260620 20:11:44.099567 16581 heartbeater.cc:503] Master 127.4.169.190:34761 was elected leader, sending a full tablet report...
I20260620 20:11:44.100221 16600 ts_tablet_manager.cc:1428] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260620 20:11:44.101307 16447 heartbeater.cc:503] Master 127.4.169.190:34761 was elected leader, sending a full tablet report...
I20260620 20:11:44.102144 16601 ts_tablet_manager.cc:1428] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 20:11:44.102840 16602 raft_consensus.cc:348] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } }
I20260620 20:11:44.103382 16602 raft_consensus.cc:374] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:11:44.103482 16602 raft_consensus.cc:729] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76a136637634f2f8b5d0f807627236c, State: Initialized, Role: FOLLOWER
I20260620 20:11:44.104279 16602 consensus_queue.cc:260] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } }
I20260620 20:11:44.107462 16602 ts_tablet_manager.cc:1428] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
W20260620 20:11:44.202353 16582 tablet.cc:2369] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:11:44.337644 16448 tablet.cc:2369] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:11:44.341928 16607 raft_consensus.cc:482] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 20:11:44.341996 16305 tablet.cc:2369] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:11:44.342216 16607 raft_consensus.cc:504] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } }
I20260620 20:11:44.344288 16607 leader_election.cc:288] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b5f6a92bd894795a7de387c69bcf511 (127.4.169.130:42945), d76a136637634f2f8b5d0f807627236c (127.4.169.129:39955)
I20260620 20:11:44.350731 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.344760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:33483 (local address 127.4.169.129:39955)
0620 20:11:44.344944 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:44.344950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:44.344973 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:11:44.345067 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:44.345075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:44.345161 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:44.345259 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:11:44.345838 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.346751 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.348806 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.349801 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.349979 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.350053 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.350319 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.350471 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:11:44.350739 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.344560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39955 (local address 127.4.169.131:33483)
0620 20:11:44.344762 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 20:11:44.344784 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:11:44.344922 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:11:44.345436 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:11:44.345442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:11:44.345475 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:11:44.345679 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.345691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.346925 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.346930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.348635 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.348655 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.349964 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.349970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.350155 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.350179 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:11:44.350283 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:11:44.350436 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 20:11:44.351493 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.345115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:44147 (local address 127.4.169.130:42945)
0620 20:11:44.345327 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:44.345336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:44.345581 (+ 245us) server_negotiation.cc:407] Connection header received
0620 20:11:44.345729 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:44.345733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:44.345822 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:44.345929 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:11:44.346879 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.347573 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.349681 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.350543 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.350723 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.350793 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.351107 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.351234 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:11:44.351552 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.344645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:42945 (local address 127.4.169.131:44147)
0620 20:11:44.345465 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 20:11:44.345492 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:11:44.345610 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:11:44.345949 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:11:44.345952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:11:44.345972 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:11:44.346723 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.346732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.347743 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.347749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.349537 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:11:44.349548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.350745 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.350752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:11:44.350923 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.350953 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:11:44.351078 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 20:11:44.351214 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":515,"threads_started":1}
I20260620 20:11:44.351864 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a75aeeb7f7478ba998204cfa7aa363" candidate_uuid: "a43bfecb31e04671a26df497be91f31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76a136637634f2f8b5d0f807627236c" is_pre_election: true
I20260620 20:11:44.352403 16259 raft_consensus.cc:2454] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a43bfecb31e04671a26df497be91f31e in term 0.
I20260620 20:11:44.352770 16402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a75aeeb7f7478ba998204cfa7aa363" candidate_uuid: "a43bfecb31e04671a26df497be91f31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5f6a92bd894795a7de387c69bcf511" is_pre_election: true
I20260620 20:11:44.353094 16469 leader_election.cc:302] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a43bfecb31e04671a26df497be91f31e, d76a136637634f2f8b5d0f807627236c; no voters:
I20260620 20:11:44.353291 16402 raft_consensus.cc:2454] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a43bfecb31e04671a26df497be91f31e in term 0.
I20260620 20:11:44.353498 16607 raft_consensus.cc:2781] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:11:44.353667 16607 raft_consensus.cc:482] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:11:44.353804 16607 raft_consensus.cc:3037] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:11:44.356240 16607 raft_consensus.cc:504] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } }
I20260620 20:11:44.356889 16607 leader_election.cc:288] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [CANDIDATE]: Term 1 election: Requested vote from peers 8b5f6a92bd894795a7de387c69bcf511 (127.4.169.130:42945), d76a136637634f2f8b5d0f807627236c (127.4.169.129:39955)
I20260620 20:11:44.357439 16402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a75aeeb7f7478ba998204cfa7aa363" candidate_uuid: "a43bfecb31e04671a26df497be91f31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5f6a92bd894795a7de387c69bcf511"
I20260620 20:11:44.357438 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a75aeeb7f7478ba998204cfa7aa363" candidate_uuid: "a43bfecb31e04671a26df497be91f31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76a136637634f2f8b5d0f807627236c"
I20260620 20:11:44.357666 16402 raft_consensus.cc:3037] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:11:44.357685 16259 raft_consensus.cc:3037] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:11:44.361476 16259 raft_consensus.cc:2454] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a43bfecb31e04671a26df497be91f31e in term 1.
I20260620 20:11:44.361527 16402 raft_consensus.cc:2454] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a43bfecb31e04671a26df497be91f31e in term 1.
I20260620 20:11:44.361981 16471 leader_election.cc:302] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5f6a92bd894795a7de387c69bcf511, a43bfecb31e04671a26df497be91f31e; no voters:
I20260620 20:11:44.362296 16607 raft_consensus.cc:2781] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:11:44.362735 16607 raft_consensus.cc:686] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 1 LEADER]: Becoming Leader. State: Replica: a43bfecb31e04671a26df497be91f31e, State: Running, Role: LEADER
I20260620 20:11:44.363245 16607 consensus_queue.cc:237] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } }
I20260620 20:11:44.368449 16090 catalog_manager.cc:5495] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e reported cstate change: term changed from 0 to 1, leader changed from <none> to a43bfecb31e04671a26df497be91f31e (127.4.169.131). New cstate: current_term: 1 leader_uuid: "a43bfecb31e04671a26df497be91f31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43bfecb31e04671a26df497be91f31e" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 } health_report { overall_health: UNKNOWN } } }
I20260620 20:11:44.433730 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.428375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.4.169.131:38977)
0620 20:11:44.428543 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:44.428548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:44.428566 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:11:44.428774 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:44.428779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:44.428855 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:44.428988 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:11:44.429384 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.430068 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.431641 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.432046 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.432153 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.432666 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:11:44.433219 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:11:44.433337 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.433455 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.433527 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:11:44.442358 16259 raft_consensus.cc:1261] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [term 1 FOLLOWER]: Refusing update from remote peer a43bfecb31e04671a26df497be91f31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:11:44.442358 16402 raft_consensus.cc:1261] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [term 1 FOLLOWER]: Refusing update from remote peer a43bfecb31e04671a26df497be91f31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:11:44.443516 16607 consensus_queue.cc:1035] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d76a136637634f2f8b5d0f807627236c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:11:44.443825 16610 consensus_queue.cc:1035] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b5f6a92bd894795a7de387c69bcf511" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:11:44.456163 16613 mvcc.cc:205] Tried to move back new op lower bound from 7299015902985986048 to 7299015902676094976. Current Snapshot: MvccSnapshot[applied={T|T < 7299015902985986048}]
I20260620 20:11:44.475020 16091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:11:44.475358 16091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:11:44.479823 16091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d287f4aa7bb948679d9824fd0d13c31a: Sending DeleteTablet for 3 replicas of tablet 29a75aeeb7f7478ba998204cfa7aa363
I20260620 20:11:44.481356 16516 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a75aeeb7f7478ba998204cfa7aa363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:11:44 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:50928
I20260620 20:11:44.481346 16382 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a75aeeb7f7478ba998204cfa7aa363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:11:44 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:47932
I20260620 20:11:44.481858 16239 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a75aeeb7f7478ba998204cfa7aa363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:11:44 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:49130
I20260620 20:11:44.484993 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 16171
I20260620 20:11:44.490432 16624 tablet_replica.cc:330] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: stopping tablet replica
I20260620 20:11:44.491021 16624 raft_consensus.cc:2229] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:11:44.491864 16624 raft_consensus.cc:2258] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:11:44.496870 16624 ts_tablet_manager.cc:1905] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:11:44.510138 16622 tablet_replica.cc:330] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: stopping tablet replica
I20260620 20:11:44.510703 16622 raft_consensus.cc:2229] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:11:44.511267 16622 raft_consensus.cc:2258] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:11:44.512753 16624 ts_tablet_manager.cc:1918] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:11:44.512990 16624 log.cc:1196] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/29a75aeeb7f7478ba998204cfa7aa363
I20260620 20:11:44.513618 16624 ts_tablet_manager.cc:1939] T 29a75aeeb7f7478ba998204cfa7aa363 P d76a136637634f2f8b5d0f807627236c: Deleting consensus metadata
I20260620 20:11:44.515025 16623 tablet_replica.cc:330] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: stopping tablet replica
I20260620 20:11:44.515511 16623 raft_consensus.cc:2229] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:11:44.516350 16075 catalog_manager.cc:4841] TS d76a136637634f2f8b5d0f807627236c (127.4.169.129:39955): tablet 29a75aeeb7f7478ba998204cfa7aa363 (table test-table [id=ab8714ab3f8a49dfab7f6fe9ed10880e]) successfully deleted
Jun 20 20:11:44 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:11:44.517185 16623 raft_consensus.cc:2258] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:11:44.519713 16622 ts_tablet_manager.cc:1905] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:11:44.519879 16623 ts_tablet_manager.cc:1905] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:11:44.527194 16623 ts_tablet_manager.cc:1918] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:11:44.527266 16622 ts_tablet_manager.cc:1918] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:11:44.527392 16623 log.cc:1196] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/29a75aeeb7f7478ba998204cfa7aa363
I20260620 20:11:44.527410 16622 log.cc:1196] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/29a75aeeb7f7478ba998204cfa7aa363
I20260620 20:11:44.527940 16623 ts_tablet_manager.cc:1939] T 29a75aeeb7f7478ba998204cfa7aa363 P a43bfecb31e04671a26df497be91f31e: Deleting consensus metadata
I20260620 20:11:44.527935 16622 ts_tablet_manager.cc:1939] T 29a75aeeb7f7478ba998204cfa7aa363 P 8b5f6a92bd894795a7de387c69bcf511: Deleting consensus metadata
I20260620 20:11:44.527967 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.486970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.4.169.129:39955)
0620 20:11:44.487348 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:44.487358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:44.500204 (+ 12846us) server_negotiation.cc:407] Connection header received
0620 20:11:44.500740 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:44.500746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:44.500829 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:44.500897 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:11:44.502988 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.504459 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.507420 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.508323 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.508457 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.518763 (+ 10306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:11:44.518783 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:11:44.518805 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:11:44.518866 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:11:44.523165 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:44.524291 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:44.524302 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:44.524306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:44.524487 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:44.524962 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:44.524968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:44.524971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:44.525465 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:11:44.525781 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.527073 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.527538 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260620 20:11:44.528985 16299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:11:44.529727 16075 catalog_manager.cc:4841] TS a43bfecb31e04671a26df497be91f31e (127.4.169.131:38977): tablet 29a75aeeb7f7478ba998204cfa7aa363 (table test-table [id=ab8714ab3f8a49dfab7f6fe9ed10880e]) successfully deleted
I20260620 20:11:44.529836 16077 catalog_manager.cc:4841] TS 8b5f6a92bd894795a7de387c69bcf511 (127.4.169.130:42945): tablet 29a75aeeb7f7478ba998204cfa7aa363 (table test-table [id=ab8714ab3f8a49dfab7f6fe9ed10880e]) successfully deleted
I20260620 20:11:44.664813 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 16171
I20260620 20:11:44.683290 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 16310
Jun 20 20:11:44 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:11:44.702775 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.684196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.4.169.130:42945)
0620 20:11:44.684369 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:44.684373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:44.684394 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:11:44.684605 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:44.684609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:44.684672 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:44.684755 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:11:44.686467 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.687306 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.690858 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.691278 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.691385 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.696642 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:11:44.696658 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:11:44.696674 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:11:44.696725 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:11:44.699656 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:44.700461 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:44.700469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:44.700472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:44.700612 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:44.701021 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:44.701028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:44.701031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:44.701437 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:11:44.701748 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.702096 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.702328 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:11:44.703500 16442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:11:44.794579 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 16310
I20260620 20:11:44.817306 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 16453
Jun 20 20:11:44 dist-test-slave-4bwq krb5kdc[15615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:11:44.835636 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:11:44.818156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.4.169.131:38977)
0620 20:11:44.818439 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 20:11:44.818445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:11:44.818587 (+ 142us) server_negotiation.cc:407] Connection header received
0620 20:11:44.818791 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:11:44.818797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:11:44.818881 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:11:44.819003 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:11:44.820247 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.821468 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.822884 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:11:44.823584 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:11:44.823754 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:11:44.828091 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:11:44.828114 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:11:44.828138 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:11:44.828183 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:11:44.832396 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:44.833250 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:44.833262 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:44.833266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:44.833460 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:11:44.833859 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:11:44.833864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:11:44.833868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:11:44.834329 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:11:44.834728 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0620 20:11:44.834991 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 20:11:44.835225 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 20:11:44.836541 16576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:11:44.970463 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 16453
I20260620 20:11:44.997285 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 16059
I20260620 20:11:44.998624 16121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:11:45.117676 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 16059
I20260620 20:11:45.139102 4774 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:11:45.475541 4774 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:11:45.475749 4774 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:11:45.822332 4774 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:11:45.822 UTC [15645] LOG: received smart shutdown request
2026-06-20 20:11:45.827 UTC [15645] LOG: background worker "logical replication launcher" (PID 15654) exited with exit code 1
2026-06-20 20:11:45.827 UTC [15649] LOG: shutting down
2026-06-20 20:11:46.026 UTC [15645] LOG: database system is shut down
2026-06-20T20:11:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51225 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:11:46 dist-test-slave-4bwq krb5kdc[16638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:11:46 dist-test-slave-4bwq krb5kdc[16638](info): set up 2 sockets
Jun 20 20:11:46 dist-test-slave-4bwq krb5kdc[16638](info): commencing operation
krb5kdc: starting...
W20260620 20:11:47.303009 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:11:47 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:11:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.169.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.169.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/rangeradmin_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/rangeradmin_127.4.169.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.169.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.169.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/rangerlookup_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/rangerlookup_127.4.169.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.169.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.169.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/HTTP_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/HTTP_127.4.169.148@KRBTEST.COM.keytab.
I20260620 20:11:47.650184 4774 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBzMLvU/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/postgres -l logfile start
2026-06-20 20:11:50.453 UTC [16668] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:11:50.453 UTC [16668] LOG: listening on IPv4 address "127.4.169.148", port 46711
2026-06-20 20:11:50.457 UTC [16668] LOG: listening on Unix socket "/tmp/.s.PGSQL.46711"
2026-06-20 20:11:50.482 UTC [16671] LOG: database system was shut down at 2026-06-20 20:11:48 UTC
2026-06-20 20:11:50.487 UTC [16668] LOG: database system is ready to accept connections
I20260620 20:11:51.458757 4774 mini_postgres.cc:96] Postgres bound to 46711
2026-06-20 20:11:51.476 UTC [16679] FATAL: database "slave" does not exist
127.4.169.148:46711 - accepting connections
I20260620 20:11:51.477514 4774 mini_ranger.cc:161] Starting Ranger...
I20260620 20:11:51.504782 4774 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:11:52.037171 4774 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:11:52.037288 4774 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-admin
2026-06-20 20:11:52,330 [I] DB FLAVOR :POSTGRES
2026-06-20 20:11:52,330 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:11:52,331 [I] Checking connection..
2026-06-20 20:11:52,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:11:52,574 [I] Checking connection passed.
2026-06-20 20:11:52,575 [I] --------- Verifying version history table ---------
2026-06-20 20:11:52,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:11:52,849 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:11:52,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:11:53,103 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:11:53,104 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:11:53,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:11:53.375 UTC [16743] WARNING: there is no transaction in progress
2026-06-20 20:11:53,381 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:11:53,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:11:53,658 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:11:53,659 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:11:53,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:11:53,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:11:54,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4bwq.c.gcp-upstream.internal','N') ;"
2026-06-20 20:11:54,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:11:54,674 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:11:54,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:11:54,910 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:11:54,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:11:55,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:11:55,451 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:11:55,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:11:56.416 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.433 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.449 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.466 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.484 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.506 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.565 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.573 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.589 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.600 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.611 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.626 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.637 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.645 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.653 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:56.660 UTC [16851] WARNING: there is no transaction in progress
2026-06-20 20:11:57,364 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:11:57,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:11:57,606 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:11:57,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:11:57,878 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:11:57,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:11:58,148 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:11:58,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:11:58,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:11:58,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4bwq.c.gcp-upstream.internal';"
2026-06-20 20:11:58,908 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:11:58,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:11:59,150 [I] DB_PATCHES have already been applied
I20260620 20:11:59.155993 4774 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBzMLvU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:11:59.156080 4774 mini_ranger.cc:193] Using host: 127.4.169.148
I20260620 20:11:59.171913 4774 mini_ranger.cc:227] Ranger admin URL: http://127.4.169.148:40221
Jun 20, 2026 8:11:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:11:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:11:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:11:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:11:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:12:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:12:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:12:00.309499 4774 mini_ranger.cc:160] Time spent starting Ranger: real 8.832s user 0.004s sys 0.039s
Jun 20, 2026 8:12:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:12:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 8:12:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:12:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:12:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:12:22.719121 4774 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.169.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.169.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/rangerkms_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.169.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/rangerkms_127.4.169.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.169.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.169.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.169.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/HTTP_127.4.169.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.169.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/HTTP_127.4.169.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:12:22 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986342, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:12:22.879055 4774 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:12:22.899756 4774 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:12:23.688097 4774 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:12:23.688213 4774 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-kms
I20260620 20:12:23.688258 4774 mini_ranger_kms.cc:227] Using postgres at 127.4.169.148:46711
2026-06-20 20:12:24,042 [I] DB FLAVOR :POSTGRES
2026-06-20 20:12:24,042 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:12:24,042 [I] Checking connection
2026-06-20 20:12:24,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:12:24,299 [I] connection success
2026-06-20 20:12:24,300 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:12:24,300 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:12:24,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:12:24,531 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:12:24,531 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:12:24,531 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:12:24,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.169.148:46711/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:12:24,781 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:12:25.284854 4774 mini_ranger_kms.cc:310] Created kms service
I20260620 20:12:25.517053 4774 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:12:25.561583 4774 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:12:25 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986342, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:25.970996 4774 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:12:25.971309 4774 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBzMLvU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBzMLvU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:12:25.971387 4774 mini_ranger_kms.cc:240] Using host: 127.4.169.148
I20260620 20:12:25.992976 4774 mini_ranger_kms.cc:276] Ranger KMS PID: 17032
I20260620 20:12:25.993108 4774 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.169.148:59317
Jun 20, 2026 8:12:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:12:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:12:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:12:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:12:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:12:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 20:12:28 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986348, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.169.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:28.267881 4774 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.389s user 0.078s sys 0.289s
I20260620 20:12:28.268150 4774 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:12:28.268347 4774 mini_ranger_kms.cc:401] 127.4.169.148:59317/kms/v1/keys
Jun 20 20:12:28 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986348, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.169.148@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
Jun 20 20:12:28 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986348, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.169.148@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
Jun 20 20:12:31 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:12:31.190290 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:35535
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:35535
--ranger_config_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:59317/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:31.527120 17082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:31.527434 17082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:31.527598 17082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:31.534970 17082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:12:31.535090 17082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:31.535184 17082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:31.535282 17082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:12:31.535460 17082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:12:31.545881 17082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.169.148:59317/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:35535
--ranger_config_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:31.547291 17082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:31.548640 17082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:31.560899 17090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:31.560887 17087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:31.560986 17088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:31.561698 17082 server_base.cc:1034] running on GCE node
I20260620 20:12:31.562637 17082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:31.564242 17082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:31.565480 17082 hybrid_clock.cc:648] HybridClock initialized: now 1781986351565438 us; error 36 us; skew 500 ppm
I20260620 20:12:31.565843 17082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:31 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986351, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:31.572937 17082 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:12:31.574605 17082 webserver.cc:460] Webserver started at http://127.4.169.190:43003/ using document root <none> and password file <none>
I20260620 20:12:31.575279 17082 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:31.575378 17082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:31.575654 17082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:12:31 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986351, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:31.634295 17082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "6f9a9ea8663741928f07626f82afd2c3"
format_stamp: "Formatted at 2026-06-20 20:12:31 on dist-test-slave-4bwq"
server_key: "37b36fc1c459c45375fbfdba682b2baf"
server_key_iv: "a15514c770ab61b5ca392b6287b81a3c"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:31.635360 17082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "6f9a9ea8663741928f07626f82afd2c3"
format_stamp: "Formatted at 2026-06-20 20:12:31 on dist-test-slave-4bwq"
server_key: "37b36fc1c459c45375fbfdba682b2baf"
server_key_iv: "a15514c770ab61b5ca392b6287b81a3c"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:31.641407 17082 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260620 20:12:31.658214 17097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:31.659991 17082 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:12:31.660202 17082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "6f9a9ea8663741928f07626f82afd2c3"
format_stamp: "Formatted at 2026-06-20 20:12:31 on dist-test-slave-4bwq"
server_key: "37b36fc1c459c45375fbfdba682b2baf"
server_key_iv: "a15514c770ab61b5ca392b6287b81a3c"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:31.660467 17082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:31.685098 17082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:31.689442 17082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:31.689776 17082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:31.709133 17082 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:35535
I20260620 20:12:31.709153 17148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:35535 every 8 connection(s)
I20260620 20:12:31.710635 17082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:12:31.714540 17149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:31.716066 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 17082
I20260620 20:12:31.716300 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:12:31.716969 4774 external_mini_cluster.cc:1428] Setting key 1d9945ebee73ee795fd1d79042010185
I20260620 20:12:31.728271 17149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Bootstrap starting.
I20260620 20:12:31.731942 17149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Neither blocks nor log segments found. Creating new log.
Jun 20 20:12:31 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:31.733052 17149 log.cc:824] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:31.736682 17149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: No bootstrap required, opened a new log
I20260620 20:12:31.742457 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:31.719188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.4.169.190:35535)
0620 20:12:31.720074 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:31.720096 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:31.720146 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:12:31.722579 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:31.722612 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:31.723132 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:31.723605 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:31.724819 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:31.727023 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:31.728513 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:31.729531 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:31.729711 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:31.733793 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:31.733802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:31.733819 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:31.733854 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:31.737715 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:31.738469 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:31.738477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:31.738486 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:31.738709 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:31.739103 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:31.739108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:31.739111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:31.739853 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:31.740161 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:31.740556 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 20:12:31.741003 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":389,"threads_started":1}
I20260620 20:12:31.746305 17149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } }
I20260620 20:12:31.746810 17149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:31.746919 17149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9a9ea8663741928f07626f82afd2c3, State: Initialized, Role: FOLLOWER
I20260620 20:12:31.747565 17149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } }
I20260620 20:12:31.747860 17149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:12:31.747984 17149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:12:31.748114 17149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:31.750219 17149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } }
I20260620 20:12:31.750744 17149 leader_election.cc:302] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f9a9ea8663741928f07626f82afd2c3; no voters:
I20260620 20:12:31.751399 17149 leader_election.cc:288] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:12:31.751531 17154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:12:31.752681 17154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6f9a9ea8663741928f07626f82afd2c3, State: Running, Role: LEADER
I20260620 20:12:31.753201 17154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } }
I20260620 20:12:31.753667 17149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:12:31.756171 17156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f9a9ea8663741928f07626f82afd2c3. Latest consensus state: current_term: 1 leader_uuid: "6f9a9ea8663741928f07626f82afd2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } } }
I20260620 20:12:31.756733 17156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:31.756218 17155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f9a9ea8663741928f07626f82afd2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } } }
I20260620 20:12:31.757135 17155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:31.759071 17163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:12:31.765451 17163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:12:31.766242 17149 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:12:31.785043 17163 catalog_manager.cc:1347] Generated new cluster ID: f80cd836ee424fb0a221fbf9c3149eee
I20260620 20:12:31.785224 17163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:12:31.812212 17163 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:12:31.813706 17163 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:12:31.838080 17163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Generated new TSK 0
I20260620 20:12:31.839090 17163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:12:32.566314 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:59317/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35535
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:12:32 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 20:12:32.960242 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:32.960597 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:32.960775 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:32.969333 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:32.969486 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:32.969663 17194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:12:32.980634 17194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.169.148:59317/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:32.982046 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:32.983494 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:33.004733 17204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:33.010057 17194 server_base.cc:1034] running on GCE node
W20260620 20:12:33.010324 17207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:33.010589 17205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:33.011397 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:33.012524 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:33.013805 17194 hybrid_clock.cc:648] HybridClock initialized: now 1781986353013721 us; error 79 us; skew 500 ppm
I20260620 20:12:33.014267 17194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:33 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986353, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:33.017778 17194 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:12:33.019721 17194 webserver.cc:460] Webserver started at http://127.4.169.129:33131/ using document root <none> and password file <none>
I20260620 20:12:33.020552 17194 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:33.020701 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:33.021041 17194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:12:33 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986353, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:33.050901 17194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f"
format_stamp: "Formatted at 2026-06-20 20:12:33 on dist-test-slave-4bwq"
server_key: "18f382e1d30ffcec54748266a98ee1d7"
server_key_iv: "9f9c7706d8f27e1711c3427a232571fa"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:33.051862 17194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f"
format_stamp: "Formatted at 2026-06-20 20:12:33 on dist-test-slave-4bwq"
server_key: "18f382e1d30ffcec54748266a98ee1d7"
server_key_iv: "9f9c7706d8f27e1711c3427a232571fa"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:33.058660 17194 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 20:12:33.095136 17214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:33.099259 17194 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.005s sys 0.008s
I20260620 20:12:33.099494 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f"
format_stamp: "Formatted at 2026-06-20 20:12:33 on dist-test-slave-4bwq"
server_key: "18f382e1d30ffcec54748266a98ee1d7"
server_key_iv: "9f9c7706d8f27e1711c3427a232571fa"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:33.099781 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:33.128167 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:33.133694 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:33.134120 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:33.146864 17194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:33.155405 17194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:33.155624 17194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:12:33.155836 17194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:33.156040 17194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:33.202898 17326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:42991 every 8 connection(s)
I20260620 20:12:33.203008 17194 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:42991
I20260620 20:12:33.204972 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:12:33.215008 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 17194
I20260620 20:12:33.215230 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:12:33.216046 4774 external_mini_cluster.cc:1428] Setting key 32d9a8cbf925d6c67e5ea84c83a4cbfd
Jun 20 20:12:33 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986353, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:12:33.248268 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:33.208217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:35377 (local address 127.4.169.190:35535)
0620 20:12:33.208712 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:33.208717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:33.222717 (+ 14000us) server_negotiation.cc:407] Connection header received
0620 20:12:33.222810 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:33.222816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:33.222919 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:33.223068 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:33.229285 (+ 6217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:33.230208 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:33.232805 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:33.233229 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:33.233383 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:33.238092 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:33.238097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:33.238100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:33.238154 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:33.241871 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:33.243052 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:33.243059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:33.243069 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:33.243211 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:33.243833 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:33.243840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:33.243844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:33.244224 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:33.244394 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:33.247596 (+ 3202us) server_negotiation.cc:299] Negotiation successful
0620 20:12:33.247858 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":284,"threads_started":1}
I20260620 20:12:33.250716 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:33.209016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:35535 (local address 127.4.169.129:35377)
0620 20:12:33.215122 (+ 6106us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:33.215187 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:33.217673 (+ 2486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:33.227366 (+ 9693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:33.227383 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:33.228080 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:33.229030 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:33.229062 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:33.230436 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:33.230442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:33.232613 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:33.232641 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:33.233486 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:33.233495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:33.233819 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:33.234734 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:33.234768 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:33.237800 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:33.242188 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:33.242196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:33.242213 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:33.242905 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:33.243358 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:33.243363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:33.243366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:33.243660 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:33.246534 (+ 2874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:33.246543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:33.247075 (+ 532us) client_negotiation.cc:769] Sending connection context
0620 20:12:33.248520 (+ 1445us) client_negotiation.cc:240] Negotiation successful
0620 20:12:33.249139 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5716,"thread_start_us":7826,"threads_started":1}
I20260620 20:12:33.253054 17327 heartbeater.cc:347] Connected to a master server at 127.4.169.190:35535
I20260620 20:12:33.253432 17327 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:33.254350 17327 heartbeater.cc:511] Master 127.4.169.190:35535 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
I20260620 20:12:33.257513 17114 ts_manager.cc:194] Registered new tserver with Master: b7e1dc2fd7dc48938c207b3d3fc8463f (127.4.169.129:42991)
I20260620 20:12:33.259732 17114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:35377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:12:33.292343 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:59317/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35535
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:12:33 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
Jun 20 20:12:33 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
W20260620 20:12:33.814934 17333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:33.815462 17333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:33.815665 17333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:33.827543 17333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:33.827790 17333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:33.828039 17333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:12:33.844027 17333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.169.148:59317/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:33.846063 17333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:33.847965 17333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:33.883219 17341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:33.887197 17340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:33.890038 17333 server_base.cc:1034] running on GCE node
W20260620 20:12:33.894739 17344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:33.895637 17333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:33.904814 17333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:33.906231 17333 hybrid_clock.cc:648] HybridClock initialized: now 1781986353906158 us; error 77 us; skew 500 ppm
I20260620 20:12:33.906764 17333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:33 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986353, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:33.911365 17333 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:12:33.932554 17333 webserver.cc:460] Webserver started at http://127.4.169.130:43681/ using document root <none> and password file <none>
I20260620 20:12:33.933743 17333 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:33.933971 17333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:33.934518 17333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:12:33 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986353, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:33.960178 17333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "e376c5712cf643838c668481f680fd7e"
format_stamp: "Formatted at 2026-06-20 20:12:33 on dist-test-slave-4bwq"
server_key: "614bb0958a3e7e84fbc14f8812292c39"
server_key_iv: "f752fa242507556a0ecbd951d3e577bf"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:33.961352 17333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "e376c5712cf643838c668481f680fd7e"
format_stamp: "Formatted at 2026-06-20 20:12:33 on dist-test-slave-4bwq"
server_key: "614bb0958a3e7e84fbc14f8812292c39"
server_key_iv: "f752fa242507556a0ecbd951d3e577bf"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:33.968565 17333 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 20:12:33.987630 17357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:33.989320 17333 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:12:33.989519 17333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "e376c5712cf643838c668481f680fd7e"
format_stamp: "Formatted at 2026-06-20 20:12:33 on dist-test-slave-4bwq"
server_key: "614bb0958a3e7e84fbc14f8812292c39"
server_key_iv: "f752fa242507556a0ecbd951d3e577bf"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:33.989791 17333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:34.024976 17333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:34.028978 17333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:34.029309 17333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:34.030292 17333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:34.031730 17333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:34.031837 17333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:34.031988 17333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:34.032075 17333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:34.066489 17333 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:36545
I20260620 20:12:34.066520 17469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:36545 every 8 connection(s)
I20260620 20:12:34.067828 17333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:12:34.070082 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 17333
I20260620 20:12:34.070267 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:12:34.070890 4774 external_mini_cluster.cc:1428] Setting key 4b619abfa01454aed1eb65a238030613
Jun 20 20:12:34 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986353, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:34.091017 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:34.070699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:35147 (local address 127.4.169.190:35535)
0620 20:12:34.071215 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:34.071222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:34.072404 (+ 1182us) server_negotiation.cc:407] Connection header received
0620 20:12:34.074545 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:34.074553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:34.074665 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:34.074792 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:34.076313 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.077597 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.079373 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.080129 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.080262 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:34.084512 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:34.084518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:34.084521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:34.084556 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:34.087579 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.088418 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.088423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.088431 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.088561 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.089038 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.089042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.089045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.089336 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:34.089573 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:34.090332 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 20:12:34.090630 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
I20260620 20:12:34.092021 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:34.071357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:35535 (local address 127.4.169.130:35147)
0620 20:12:34.072218 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:34.072276 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:34.074179 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:34.075003 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:34.075011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:34.075522 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:34.076123 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:34.076141 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.077772 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.077776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:34.079214 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:34.079234 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.080425 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.080434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:34.080771 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:34.081576 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:34.081611 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:34.084257 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:34.087766 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:34.087776 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:34.087800 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:34.088264 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:34.088680 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:34.088685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:34.088688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:34.088898 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:34.089545 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:34.089556 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:34.089944 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 20:12:34.090225 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 20:12:34.090810 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":349,"threads_started":1}
I20260620 20:12:34.093699 17470 heartbeater.cc:347] Connected to a master server at 127.4.169.190:35535
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:12:34.094076 17470 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:34.095027 17470 heartbeater.cc:511] Master 127.4.169.190:35535 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
I20260620 20:12:34.096644 17114 ts_manager.cc:194] Registered new tserver with Master: e376c5712cf643838c668481f680fd7e (127.4.169.130:36545)
I20260620 20:12:34.097461 17114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:35147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:12:34.126194 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:59317/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35535
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:12:34.263947 17327 heartbeater.cc:503] Master 127.4.169.190:35535 was elected leader, sending a full tablet report...
W20260620 20:12:34.458720 17476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:34.459017 17476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:34.459152 17476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:34.466770 17476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:34.466905 17476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:34.467018 17476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:12:34.476996 17476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.169.148:59317/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:34.478278 17476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:34.479625 17476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:34.494508 17481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:34.494613 17482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:34.494585 17484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:34.495826 17476 server_base.cc:1034] running on GCE node
I20260620 20:12:34.496495 17476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:34.497598 17476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:34.498833 17476 hybrid_clock.cc:648] HybridClock initialized: now 1781986354498776 us; error 58 us; skew 500 ppm
I20260620 20:12:34.499154 17476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:34 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986354, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:34.502663 17476 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:12:34.504261 17476 webserver.cc:460] Webserver started at http://127.4.169.131:42701/ using document root <none> and password file <none>
I20260620 20:12:34.505041 17476 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:34.505175 17476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:34.505494 17476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:12:34 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986354, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:34.521180 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "13b79632d31c44d8b62e38b64d84455f"
format_stamp: "Formatted at 2026-06-20 20:12:34 on dist-test-slave-4bwq"
server_key: "2b51257730c39885145c0d7d21f40f09"
server_key_iv: "3b6acd4c885401918a43b2dffd5ffcf4"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:34.522212 17476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "13b79632d31c44d8b62e38b64d84455f"
format_stamp: "Formatted at 2026-06-20 20:12:34 on dist-test-slave-4bwq"
server_key: "2b51257730c39885145c0d7d21f40f09"
server_key_iv: "3b6acd4c885401918a43b2dffd5ffcf4"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:34.528569 17476 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 20:12:34.543562 17491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:34.545123 17476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:12:34.545331 17476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "13b79632d31c44d8b62e38b64d84455f"
format_stamp: "Formatted at 2026-06-20 20:12:34 on dist-test-slave-4bwq"
server_key: "2b51257730c39885145c0d7d21f40f09"
server_key_iv: "3b6acd4c885401918a43b2dffd5ffcf4"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:34.545575 17476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:34.562733 17476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:34.567026 17476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:34.567293 17476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:34.568270 17476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:34.569686 17476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:34.569761 17476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:34.569847 17476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:34.569891 17476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:34.602066 17476 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:33583
I20260620 20:12:34.602087 17603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:33583 every 8 connection(s)
I20260620 20:12:34.603461 17476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:12:34.609287 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 17476
I20260620 20:12:34.609462 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:12:34.610076 4774 external_mini_cluster.cc:1428] Setting key 017b0f5d1ae9b2af3e7627570bde2523
Jun 20 20:12:34 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986354, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:34.623662 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:34.606191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:34983 (local address 127.4.169.190:35535)
0620 20:12:34.606753 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:34.606758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:34.607826 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 20:12:34.609702 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:34.609709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:34.609836 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:34.609972 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:34.611834 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.612732 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.614589 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.615005 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.615103 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:34.618359 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:34.618364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:34.618367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:34.618456 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:34.620852 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.621538 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.621542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.621544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.621706 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.622111 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.622114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.622116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.622373 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:34.622559 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:34.623183 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 20:12:34.623342 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":293,"threads_started":1}
I20260620 20:12:34.624605 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:34.606847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:35535 (local address 127.4.169.131:34983)
0620 20:12:34.607660 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:34.607700 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:34.609340 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:34.610286 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:34.610299 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:34.610900 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:34.611653 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:34.611676 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.612901 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.612907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:34.614374 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:34.614442 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.615192 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.615197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:34.615403 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:34.616111 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:34.616134 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:34.618142 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:34.620978 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:34.620985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:34.621005 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:34.621431 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:34.621795 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:34.621798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:34.621800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:34.621986 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:34.622549 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:34.622554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:34.622856 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 20:12:34.623099 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 20:12:34.623468 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":280,"threads_started":1}
I20260620 20:12:34.626051 17604 heartbeater.cc:347] Connected to a master server at 127.4.169.190:35535
I20260620 20:12:34.626353 17604 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:34.627086 17604 heartbeater.cc:511] Master 127.4.169.190:35535 requested a full tablet report, sending...
I20260620 20:12:34.628486 17113 ts_manager.cc:194] Registered new tserver with Master: 13b79632d31c44d8b62e38b64d84455f (127.4.169.131:33583)
I20260620 20:12:34.629395 17113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:34983
I20260620 20:12:34.638756 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:12:34.640442 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 17194
Jun 20 20:12:34 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:12:34.657356 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:34.641384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.4.169.129:42991)
0620 20:12:34.641991 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:34.642033 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:34.642065 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:12:34.642138 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:34.642144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:34.642350 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:34.642571 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:34.644084 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.645097 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.646644 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.647281 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.647432 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:34.651999 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:34.652010 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:34.652023 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:34.652065 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:34.654905 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.655573 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.655580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.655582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.655717 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.656026 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.656029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.656031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.656344 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:34.656576 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:34.656857 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 20:12:34.657075 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":276,"threads_started":1}
I20260620 20:12:34.658735 17322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:34.742304 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 17194
I20260620 20:12:34.758097 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 17333
Jun 20 20:12:34 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:12:34.778990 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:34.759211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.4.169.130:36545)
0620 20:12:34.759707 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:34.759715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:34.759740 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:12:34.759829 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:34.759837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:34.760084 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:34.760220 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:34.761452 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.762524 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.763958 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.764629 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.764730 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:34.773327 (+ 8597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:34.773342 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:34.773358 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:34.773429 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:34.776393 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.777090 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.777097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.777099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.777239 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.777576 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.777579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.777581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.777910 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:34.778125 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:34.778458 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:12:34.778698 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":262,"threads_started":1}
I20260620 20:12:34.780233 17465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:34.861348 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 17333
I20260620 20:12:34.876708 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 17476
Jun 20 20:12:34 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:12:34.893332 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:34.877743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.4.169.131:33583)
0620 20:12:34.878469 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:34.878476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:34.878506 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:12:34.878564 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:34.878572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:34.878785 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:34.878919 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:34.880058 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.881056 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.882241 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:34.882903 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:34.882997 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:34.887322 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:34.887337 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:34.887352 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:34.887419 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:34.890477 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.891226 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.891235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.891243 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.891373 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:34.891748 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:34.891755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:34.891758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:34.892102 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:34.892427 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:34.892713 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 20:12:34.892972 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":375,"threads_started":1}
I20260620 20:12:34.894745 17599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:34.978314 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 17476
I20260620 20:12:34.993602 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 17082
I20260620 20:12:34.994935 17144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:35.110034 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 17082
I20260620 20:12:35.135495 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:35535
--webserver_interface=127.4.169.190
--webserver_port=43003
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:35535
--ranger_config_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:59317/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.169.148:59317/kms with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:35.487133 17616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:35.487541 17616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:35.487713 17616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:35.495286 17616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:12:35.495656 17616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:35.495806 17616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:35.495914 17616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:12:35.496012 17616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:12:35.506737 17616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.169.148:59317/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:35535
--ranger_config_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=43003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:35.508205 17616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:35.509663 17616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:35.522207 17621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:35.522171 17622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:35.522250 17624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:35.523079 17616 server_base.cc:1034] running on GCE node
I20260620 20:12:35.523903 17616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:35.525487 17616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:35.526717 17616 hybrid_clock.cc:648] HybridClock initialized: now 1781986355526680 us; error 64 us; skew 500 ppm
I20260620 20:12:35.527103 17616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:35 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986355, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:35.530159 17616 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:12:35.531667 17616 webserver.cc:460] Webserver started at http://127.4.169.190:43003/ using document root <none> and password file <none>
I20260620 20:12:35.532348 17616 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:35.532492 17616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:12:35 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986355, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:35.582868 17616 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 20:12:35.586170 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:35.587708 17616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:12:35.587908 17616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "6f9a9ea8663741928f07626f82afd2c3"
format_stamp: "Formatted at 2026-06-20 20:12:31 on dist-test-slave-4bwq"
server_key: "37b36fc1c459c45375fbfdba682b2baf"
server_key_iv: "a15514c770ab61b5ca392b6287b81a3c"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:35.588896 17616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:35.620070 17616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:35.624080 17616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:35.624380 17616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:35.643404 17616 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:35535
I20260620 20:12:35.643425 17683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:35535 every 8 connection(s)
I20260620 20:12:35.645027 17616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:12:35.649006 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 17616
I20260620 20:12:35.650038 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:59317/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:42991
--local_ip_for_outbound_sockets=127.4.169.129
--tserver_master_addrs=127.4.169.190:35535
--webserver_port=33131
--webserver_interface=127.4.169.129
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.169.148:59317/kms with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:12:35.653705 17684 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:12:35.659015 17684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Bootstrap starting.
I20260620 20:12:35.678278 17684 log.cc:824] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:35.687180 17684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:12:35.687721 17684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Bootstrap complete.
I20260620 20:12:35.697034 17684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } }
I20260620 20:12:35.697966 17684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9a9ea8663741928f07626f82afd2c3, State: Initialized, Role: FOLLOWER
I20260620 20:12:35.698632 17684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } }
I20260620 20:12:35.698860 17684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:12:35.698988 17684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:12:35.699191 17684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:12:35.703161 17684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } }
I20260620 20:12:35.703889 17684 leader_election.cc:302] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f9a9ea8663741928f07626f82afd2c3; no voters:
I20260620 20:12:35.704708 17684 leader_election.cc:288] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:12:35.704843 17688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:12:35.706064 17688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [term 2 LEADER]: Becoming Leader. State: Replica: 6f9a9ea8663741928f07626f82afd2c3, State: Running, Role: LEADER
I20260620 20:12:35.706792 17688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } }
I20260620 20:12:35.707276 17684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:12:35.709745 17689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6f9a9ea8663741928f07626f82afd2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } } }
I20260620 20:12:35.710361 17689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:35.709405 17690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f9a9ea8663741928f07626f82afd2c3. Latest consensus state: current_term: 2 leader_uuid: "6f9a9ea8663741928f07626f82afd2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a9ea8663741928f07626f82afd2c3" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35535 } } }
I20260620 20:12:35.711064 17690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:35.723627 17694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:12:35.731532 17694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:12:35.733590 17694 catalog_manager.cc:1259] Loaded cluster ID: f80cd836ee424fb0a221fbf9c3149eee
I20260620 20:12:35.733788 17694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:12:35.735530 17684 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:12:35.738372 17694 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:12:35.740415 17694 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Loaded TSK: 0
I20260620 20:12:35.742465 17694 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:12:36.116112 17686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:36.116643 17686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:36.116878 17686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:36.129498 17686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:36.129721 17686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:36.129948 17686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:12:36.144510 17686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.169.148:59317/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:42991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=33131
--tserver_master_addrs=127.4.169.190:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:36.146279 17686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:36.148295 17686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:36.183471 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:36.183979 17722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:36.183868 17686 server_base.cc:1034] running on GCE node
W20260620 20:12:36.185214 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:36.186215 17686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:36.187647 17686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:36.188951 17686 hybrid_clock.cc:648] HybridClock initialized: now 1781986356188888 us; error 57 us; skew 500 ppm
I20260620 20:12:36.189374 17686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:36 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:36.196285 17686 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:12:36.199962 17686 webserver.cc:460] Webserver started at http://127.4.169.129:33131/ using document root <none> and password file <none>
I20260620 20:12:36.200969 17686 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:36.201208 17686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:12:36 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:36.247632 17686 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.003s sys 0.009s
I20260620 20:12:36.263083 17732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:36.265244 17686 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.009s
I20260620 20:12:36.265496 17686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f"
format_stamp: "Formatted at 2026-06-20 20:12:33 on dist-test-slave-4bwq"
server_key: "18f382e1d30ffcec54748266a98ee1d7"
server_key_iv: "9f9c7706d8f27e1711c3427a232571fa"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:36.266764 17686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:36.307567 17686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:36.313393 17686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:36.313810 17686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:36.323837 17686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:36.325882 17686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:36.326017 17686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:36.326174 17686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:36.326254 17686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:36.422910 17686 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:42991
I20260620 20:12:36.422973 17845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:42991 every 8 connection(s)
I20260620 20:12:36.424611 17686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:12:36.433770 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 17686
I20260620 20:12:36.434990 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:59317/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:36545
--local_ip_for_outbound_sockets=127.4.169.130
--tserver_master_addrs=127.4.169.190:35535
--webserver_port=43681
--webserver_interface=127.4.169.130
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.169.148:59317/kms with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:12:36 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:36.482641 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:36.428726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:35535 (local address 127.4.169.129:60643)
0620 20:12:36.430066 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:36.430121 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:36.444186 (+ 14065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:36.447756 (+ 3570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:36.447770 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:36.448449 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:36.449317 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:36.449346 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:36.451860 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:36.451869 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:36.454097 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:36.454118 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:36.455696 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:36.455707 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:36.456056 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:36.456960 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:36.457007 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:36.460095 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:36.474550 (+ 14455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:36.474561 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:36.474582 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:36.475419 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:36.478710 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:36.478716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:36.478719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:36.479027 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:36.480124 (+ 1097us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:36.480132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:36.480515 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 20:12:36.480796 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 20:12:36.481357 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":787,"threads_started":1}
I20260620 20:12:36.486280 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:36.428738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:60643 (local address 127.4.169.190:35535)
0620 20:12:36.445226 (+ 16488us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:36.445249 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:36.445305 (+ 56us) server_negotiation.cc:407] Connection header received
0620 20:12:36.445798 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:36.445823 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:36.446419 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:36.447582 (+ 1163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:36.449584 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:36.451641 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:36.454281 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:36.455419 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:36.455632 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:36.460451 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:36.460465 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:36.460492 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:36.460561 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:36.472997 (+ 12436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:36.478203 (+ 5206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:36.478216 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:36.478226 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:36.478527 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:36.479180 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:36.479185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:36.479188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:36.479845 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:36.480039 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:36.484611 (+ 4572us) server_negotiation.cc:299] Negotiation successful
0620 20:12:36.485056 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16163,"thread_start_us":15897,"threads_started":1}
I20260620 20:12:36.490650 17846 heartbeater.cc:347] Connected to a master server at 127.4.169.190:35535
I20260620 20:12:36.490996 17846 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:36.491885 17846 heartbeater.cc:511] Master 127.4.169.190:35535 requested a full tablet report, sending...
W20260620 20:12:36.495316 17846 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:35535 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:12:36.495551 17846 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:36.495867 17846 heartbeater.cc:511] Master 127.4.169.190:35535 requested a full tablet report, sending...
I20260620 20:12:36.497255 17846 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:36.497609 17846 heartbeater.cc:511] Master 127.4.169.190:35535 requested a full tablet report, sending...
W20260620 20:12:36.498783 17846 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:12:36.895248 17850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:36.895746 17850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:36.895942 17850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:36.907900 17850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:36.908054 17850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:36.908288 17850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:12:36.924299 17850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.169.148:59317/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:36545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=43681
--tserver_master_addrs=127.4.169.190:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:36.926157 17850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:36.927968 17850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:36.976197 17865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:36.978434 17868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:36.978953 17850 server_base.cc:1034] running on GCE node
W20260620 20:12:36.980149 17864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:36.981148 17850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:36.986027 17850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:36.987349 17850 hybrid_clock.cc:648] HybridClock initialized: now 1781986356987245 us; error 92 us; skew 500 ppm
I20260620 20:12:36.987890 17850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:36 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:36.998235 17850 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:12:37.009958 17850 webserver.cc:460] Webserver started at http://127.4.169.130:43681/ using document root <none> and password file <none>
I20260620 20:12:37.011351 17850 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:37.011629 17850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:12:37 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:37.064143 17850 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:12:37.067334 17877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:37.068987 17850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:12:37.069180 17850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "e376c5712cf643838c668481f680fd7e"
format_stamp: "Formatted at 2026-06-20 20:12:33 on dist-test-slave-4bwq"
server_key: "614bb0958a3e7e84fbc14f8812292c39"
server_key_iv: "f752fa242507556a0ecbd951d3e577bf"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:37.070238 17850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:37.098862 17850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Jun 20 20:12:37 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986357, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:37.103322 17850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:37.103667 17850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:37.104923 17850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:37.108542 17850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:37.108702 17850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 20:12:37.108839 17850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:37.108937 17850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:37.183394 17850 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:36545
I20260620 20:12:37.183487 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:36545 every 8 connection(s)
I20260620 20:12:37.185310 17850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:12:37.186218 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 17850
I20260620 20:12:37.187081 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.169.148:59317/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:33583
--local_ip_for_outbound_sockets=127.4.169.131
--tserver_master_addrs=127.4.169.190:35535
--webserver_port=42701
--webserver_interface=127.4.169.131
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.169.148:59317/kms with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:12:37 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:37.246356 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:37.205768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:35535 (local address 127.4.169.130:55445)
0620 20:12:37.219561 (+ 13793us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:37.219637 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:37.222320 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:37.223560 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:37.223573 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:37.224272 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:37.226004 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:37.226035 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.228238 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.228244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:37.230707 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:37.230724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.232052 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.232061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:37.232369 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:37.234032 (+ 1663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:37.234068 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:37.237017 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:37.241472 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:37.241483 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:37.241503 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:37.242127 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:37.242626 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:37.242632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:37.242636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:37.242909 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:37.243630 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:37.243639 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:37.244087 (+ 448us) client_negotiation.cc:769] Sending connection context
0620 20:12:37.244380 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 20:12:37.244956 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13432,"thread_start_us":13198,"threads_started":1}
I20260620 20:12:37.247594 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:37.206346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:55445 (local address 127.4.169.190:35535)
0620 20:12:37.218881 (+ 12535us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:37.218889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:37.222522 (+ 3633us) server_negotiation.cc:407] Connection header received
0620 20:12:37.222831 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:37.222838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:37.222936 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:37.223080 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:37.226247 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.227474 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.230903 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.231557 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.231668 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:37.237313 (+ 5645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:37.237324 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:37.237332 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:37.237389 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:37.241263 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:37.242281 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:37.242287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:37.242291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:37.242496 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:37.243045 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:37.243051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:37.243054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:37.243408 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:37.243550 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:37.246960 (+ 3410us) server_negotiation.cc:299] Negotiation successful
0620 20:12:37.247206 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12375,"thread_start_us":12295,"threads_started":1}
I20260620 20:12:37.249608 17991 heartbeater.cc:347] Connected to a master server at 127.4.169.190:35535
I20260620 20:12:37.249943 17991 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:37.250931 17991 heartbeater.cc:511] Master 127.4.169.190:35535 requested a full tablet report, sending...
I20260620 20:12:37.253094 17649 ts_manager.cc:194] Registered new tserver with Master: e376c5712cf643838c668481f680fd7e (127.4.169.130:36545)
I20260620 20:12:37.255203 17649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:55445
I20260620 20:12:37.498943 17846 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:37.499444 17846 heartbeater.cc:511] Master 127.4.169.190:35535 requested a full tablet report, sending...
I20260620 20:12:37.500948 17649 ts_manager.cc:194] Registered new tserver with Master: b7e1dc2fd7dc48938c207b3d3fc8463f (127.4.169.129:42991)
I20260620 20:12:37.502276 17649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:60643
Jun 20 20:12:37 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986357, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
W20260620 20:12:37.629362 17992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:37.629864 17992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:37.630064 17992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:37.642050 17992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:37.642263 17992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:37.642532 17992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:12:37.658025 17992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.169.148:59317/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:33583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=42701
--tserver_master_addrs=127.4.169.190:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:37.659972 17992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:37.661700 17992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:37.693086 18000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:37.693153 18001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:37.702314 17992 server_base.cc:1034] running on GCE node
W20260620 20:12:37.702343 18003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:37.703231 17992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:37.704291 17992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:37.705555 17992 hybrid_clock.cc:648] HybridClock initialized: now 1781986357705481 us; error 66 us; skew 500 ppm
I20260620 20:12:37.705924 17992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:37 dist-test-slave-4bwq krb5kdc[16638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986357, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:37.709471 17992 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:12:37.711191 17992 webserver.cc:460] Webserver started at http://127.4.169.131:42701/ using document root <none> and password file <none>
I20260620 20:12:37.711930 17992 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:37.712073 17992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:12:37 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986357, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:37.734079 17992 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:12:37.738116 18011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:37.740051 17992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:12:37.740237 17992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "13b79632d31c44d8b62e38b64d84455f"
format_stamp: "Formatted at 2026-06-20 20:12:34 on dist-test-slave-4bwq"
server_key: "2b51257730c39885145c0d7d21f40f09"
server_key_iv: "3b6acd4c885401918a43b2dffd5ffcf4"
server_key_version: "kuduclusterkey@0"
I20260620 20:12:37.741649 17992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:37.774031 17992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:37.779225 17992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:37.779536 17992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:37.780783 17992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:37.782554 17992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:37.782663 17992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:37.782761 17992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:37.782831 17992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:37.842759 17992 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:33583
I20260620 20:12:37.842832 18123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:33583 every 8 connection(s)
I20260620 20:12:37.844761 17992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:12:37.851297 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 17992
Jun 20 20:12:37 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986357, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:37.896487 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:37.852784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.4.169.190:35535)
0620 20:12:37.871125 (+ 18341us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:37.871131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:37.871147 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:12:37.871198 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:37.871203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:37.871270 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:37.873587 (+ 2317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:37.873811 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.875007 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.877260 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.877685 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.877829 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:37.884620 (+ 6791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:37.884626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:37.884630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:37.884668 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:37.891354 (+ 6686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:37.893077 (+ 1723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:37.893084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:37.893087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:37.893387 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:37.894135 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:37.894141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:37.894144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:37.894560 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:37.894849 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:37.895856 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0620 20:12:37.896097 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18238,"thread_start_us":17259,"threads_started":1}
Jun 20 20:12:37 dist-test-slave-4bwq krb5kdc[16638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986357, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for HTTP/127.4.169.148@KRBTEST.COM
I20260620 20:12:37.907267 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:37.850665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:39517 (local address 127.4.169.190:35535)
0620 20:12:37.870299 (+ 19634us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:37.870306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:37.870326 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:12:37.870450 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:37.870455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:37.870537 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:37.870642 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:37.879212 (+ 8570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.880496 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.883007 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.883775 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.883882 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:37.897475 (+ 13593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:37.897482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:37.897487 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:37.897526 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:37.903180 (+ 5654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:37.904230 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:37.904236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:37.904240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:37.904403 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:37.904968 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:37.904974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:37.904977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:37.905312 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:37.905455 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:37.906552 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0620 20:12:37.906807 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19502,"thread_start_us":20210,"threads_started":1}
I20260620 20:12:37.909806 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:37.850118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:35535 (local address 127.4.169.131:39517)
0620 20:12:37.867096 (+ 16978us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:37.867159 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:37.869555 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:37.875517 (+ 5962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:37.875531 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:37.876134 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:37.878825 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:37.878856 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.880726 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.880731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:37.882819 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:37.882835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.884252 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.884260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:37.885490 (+ 1230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:37.892156 (+ 6666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:37.892237 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:37.895549 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:37.903402 (+ 7853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:37.903413 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:37.903434 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:37.904084 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:37.904555 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:37.904560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:37.904564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:37.904835 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:37.905547 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:37.905555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:37.906041 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 20:12:37.907652 (+ 1611us) client_negotiation.cc:240] Negotiation successful
0620 20:12:37.908245 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16639,"thread_start_us":16404,"threads_started":1}
I20260620 20:12:37.911605 18124 heartbeater.cc:347] Connected to a master server at 127.4.169.190:35535
I20260620 20:12:37.911923 18124 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:37.912763 18124 heartbeater.cc:511] Master 127.4.169.190:35535 requested a full tablet report, sending...
I20260620 20:12:37.914469 17649 ts_manager.cc:194] Registered new tserver with Master: 13b79632d31c44d8b62e38b64d84455f (127.4.169.131:33583)
I20260620 20:12:37.915627 17649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:39517
I20260620 20:12:37.920507 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:12:37.961047 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:37.939083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.4.169.190:35535)
0620 20:12:37.939308 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:37.939315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:37.940140 (+ 825us) server_negotiation.cc:407] Connection header received
0620 20:12:37.940437 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:37.940444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:37.940535 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:37.940617 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:37.942960 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.944047 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.946158 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:37.946902 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:37.947010 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:37.950703 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:37.950709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:37.950712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:37.950748 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:37.954355 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:37.958560 (+ 4205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:37.958566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:37.958570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:37.958737 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:37.959229 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:37.959235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:37.959239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:37.959644 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:37.959873 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:37.960459 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 20:12:37.960702 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:12:37.969151 17649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:12:37.981429 17649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:12:38.059475 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.012724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.4.169.129:42991)
0620 20:12:38.033270 (+ 20546us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:38.033280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:38.041798 (+ 8518us) server_negotiation.cc:407] Connection header received
0620 20:12:38.042005 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:38.042014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:38.042327 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:38.042574 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:38.050670 (+ 8096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.052485 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.056181 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.057870 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.057980 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.058232 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:38.058954 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 20:12:38.059151 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20351,"thread_start_us":20576,"threads_started":1}
I20260620 20:12:38.060789 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.012167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42991 (local address 127.0.0.1:41346)
0620 20:12:38.041656 (+ 29489us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:38.041676 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:38.041816 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:38.045986 (+ 4170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:38.045991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:38.046026 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:38.050379 (+ 4353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.050415 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.052740 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.052748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.055950 (+ 3202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.055974 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.058445 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.058452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.058658 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.058713 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 20:12:38.059983 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0620 20:12:38.060218 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29366,"mutex_wait_us":1,"thread_start_us":26946,"threads_started":1}
I20260620 20:12:38.063563 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.002531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:33583 (local address 127.0.0.1:39518)
0620 20:12:38.038572 (+ 36041us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:38.038636 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:38.038901 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:38.043564 (+ 4663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:38.043571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:38.043603 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:38.044011 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.044026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.046091 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.046097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.057981 (+ 11884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.057996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.062626 (+ 4630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.062635 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.062902 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.062955 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 20:12:38.063049 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 20:12:38.063181 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35647,"mutex_wait_us":2285,"thread_start_us":44003,"threads_started":1}
I20260620 20:12:38.063637 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.003228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.4.169.131:33583)
0620 20:12:38.042594 (+ 39366us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:38.042605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:38.042641 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:12:38.042733 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:38.042740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:38.043004 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:38.043209 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:38.044243 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.045886 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.058202 (+ 12316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.059790 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.059935 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.060174 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:38.063109 (+ 2935us) server_negotiation.cc:299] Negotiation successful
0620 20:12:38.063283 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38977,"thread_start_us":39166,"threads_started":1}
I20260620 20:12:38.069164 17781 tablet_service.cc:1467] Processing CreateTablet for tablet af093a7eefac4628a864fc49ba9c3822 (DEFAULT_TABLE table=test-table [id=8e27bb49299d4d09806ce54ebcf4de99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:12:38.071202 17781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af093a7eefac4628a864fc49ba9c3822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:38.083719 18059 tablet_service.cc:1467] Processing CreateTablet for tablet af093a7eefac4628a864fc49ba9c3822 (DEFAULT_TABLE table=test-table [id=8e27bb49299d4d09806ce54ebcf4de99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:12:38.085888 18059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af093a7eefac4628a864fc49ba9c3822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:38.099169 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.050826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:36545 (local address 127.0.0.1:45682)
0620 20:12:38.061580 (+ 10754us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:38.061606 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:38.061745 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:38.085742 (+ 23997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:38.085748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:38.085779 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:38.086012 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.086025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.090524 (+ 4499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.090532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.092443 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.092465 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.098230 (+ 5765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.098238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.098521 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.098562 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:12:38.098642 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:12:38.098842 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10603,"thread_start_us":28500,"threads_started":1}
I20260620 20:12:38.100550 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.051286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.4.169.130:36545)
0620 20:12:38.085072 (+ 33786us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:38.085081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:38.085115 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:12:38.085215 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:38.085223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:38.085492 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:38.086204 (+ 712us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:38.086476 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.088014 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.092699 (+ 4685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.094138 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.099678 (+ 5540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.099928 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:38.100033 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 20:12:38.100220 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33591,"thread_start_us":47165,"threads_started":1}
I20260620 20:12:38.103286 18146 tablet_bootstrap.cc:492] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: Bootstrap starting.
I20260620 20:12:38.106233 17925 tablet_service.cc:1467] Processing CreateTablet for tablet af093a7eefac4628a864fc49ba9c3822 (DEFAULT_TABLE table=test-table [id=8e27bb49299d4d09806ce54ebcf4de99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:12:38.108126 17925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af093a7eefac4628a864fc49ba9c3822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:38.111263 18146 tablet_bootstrap.cc:654] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:38.112615 18146 log.cc:824] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:38.122252 18147 tablet_bootstrap.cc:492] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: Bootstrap starting.
I20260620 20:12:38.127476 18146 tablet_bootstrap.cc:492] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: No bootstrap required, opened a new log
I20260620 20:12:38.132452 18146 ts_tablet_manager.cc:1397] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.006s
I20260620 20:12:38.143534 18147 tablet_bootstrap.cc:654] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:38.145572 18147 log.cc:824] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:38.149528 18149 tablet_bootstrap.cc:492] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: Bootstrap starting.
I20260620 20:12:38.151593 18147 tablet_bootstrap.cc:492] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: No bootstrap required, opened a new log
I20260620 20:12:38.151911 18147 ts_tablet_manager.cc:1397] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.002s
I20260620 20:12:38.154551 18146 raft_consensus.cc:348] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } }
I20260620 20:12:38.155095 18146 raft_consensus.cc:374] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:38.155231 18146 raft_consensus.cc:729] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e1dc2fd7dc48938c207b3d3fc8463f, State: Initialized, Role: FOLLOWER
I20260620 20:12:38.155714 18149 tablet_bootstrap.cc:654] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:38.155966 18146 consensus_queue.cc:260] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } }
I20260620 20:12:38.157460 18149 log.cc:824] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:38.163694 18147 raft_consensus.cc:348] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } }
I20260620 20:12:38.164203 18147 raft_consensus.cc:374] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:38.164304 18147 raft_consensus.cc:729] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b79632d31c44d8b62e38b64d84455f, State: Initialized, Role: FOLLOWER
I20260620 20:12:38.165087 18147 consensus_queue.cc:260] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } }
I20260620 20:12:38.168109 17846 heartbeater.cc:503] Master 127.4.169.190:35535 was elected leader, sending a full tablet report...
I20260620 20:12:38.168334 18147 ts_tablet_manager.cc:1428] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: Time spent starting tablet: real 0.016s user 0.000s sys 0.014s
I20260620 20:12:38.168785 18146 ts_tablet_manager.cc:1428] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: Time spent starting tablet: real 0.036s user 0.020s sys 0.007s
I20260620 20:12:38.169454 18124 heartbeater.cc:503] Master 127.4.169.190:35535 was elected leader, sending a full tablet report...
I20260620 20:12:38.170696 18149 tablet_bootstrap.cc:492] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: No bootstrap required, opened a new log
I20260620 20:12:38.171063 18149 ts_tablet_manager.cc:1397] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.000s
I20260620 20:12:38.181706 18149 raft_consensus.cc:348] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } }
I20260620 20:12:38.182210 18149 raft_consensus.cc:374] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:38.182438 18149 raft_consensus.cc:729] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e376c5712cf643838c668481f680fd7e, State: Initialized, Role: FOLLOWER
W20260620 20:12:38.183450 17847 tablet.cc:2369] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:12:38.183400 18149 consensus_queue.cc:260] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } }
I20260620 20:12:38.185072 17991 heartbeater.cc:503] Master 127.4.169.190:35535 was elected leader, sending a full tablet report...
I20260620 20:12:38.185799 18149 ts_tablet_manager.cc:1428] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260620 20:12:38.235705 17993 tablet.cc:2369] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:12:38.365366 18125 tablet.cc:2369] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:12:38.558485 18160 raft_consensus.cc:482] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:12:38.558872 18160 raft_consensus.cc:504] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } }
I20260620 20:12:38.560480 18160 leader_election.cc:288] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7e1dc2fd7dc48938c207b3d3fc8463f (127.4.169.129:42991), e376c5712cf643838c668481f680fd7e (127.4.169.130:36545)
I20260620 20:12:38.569741 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.560931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:34379 (local address 127.4.169.130:36545)
0620 20:12:38.561161 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:38.561170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:38.561846 (+ 676us) server_negotiation.cc:407] Connection header received
0620 20:12:38.562501 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:38.562508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:38.562596 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:38.562688 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:38.563709 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.564745 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.567277 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.568579 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.568746 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.568826 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:38.569324 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 20:12:38.569464 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:12:38.569782 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.560943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:35139 (local address 127.4.169.129:42991)
0620 20:12:38.562061 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:38.562074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:38.562223 (+ 149us) server_negotiation.cc:407] Connection header received
0620 20:12:38.562402 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:38.562429 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:38.562527 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:38.562648 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:38.563324 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.564490 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.567280 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.568576 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.568746 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.568826 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:38.569324 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 20:12:38.569477 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":848,"threads_started":1}
I20260620 20:12:38.569907 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.560844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42991 (local address 127.4.169.131:35139)
0620 20:12:38.562106 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:38.562119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:38.562232 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:38.562713 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:38.562719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:38.562764 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:38.563116 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.563128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.564636 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.564641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.567107 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.567123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.568785 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.568790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.569120 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.569179 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:12:38.569281 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:12:38.569453 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"mutex_wait_us":187,"spinlock_wait_cycles":6656,"thread_start_us":1048,"threads_started":1}
I20260620 20:12:38.569947 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.560767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:36545 (local address 127.4.169.131:34379)
0620 20:12:38.561712 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:38.561739 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:38.562347 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:38.562713 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:38.562719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:38.563169 (+ 450us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:38.563553 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.563561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.564900 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.564905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.567107 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:38.567123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.568780 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.568787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:38.569125 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.569178 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 20:12:38.569286 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 20:12:38.569453 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":146,"spinlock_wait_cycles":896,"thread_start_us":571,"threads_started":1}
I20260620 20:12:38.571277 17801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af093a7eefac4628a864fc49ba9c3822" candidate_uuid: "13b79632d31c44d8b62e38b64d84455f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" is_pre_election: true
I20260620 20:12:38.571277 17945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af093a7eefac4628a864fc49ba9c3822" candidate_uuid: "13b79632d31c44d8b62e38b64d84455f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e376c5712cf643838c668481f680fd7e" is_pre_election: true
I20260620 20:12:38.571897 17945 raft_consensus.cc:2454] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b79632d31c44d8b62e38b64d84455f in term 0.
I20260620 20:12:38.571900 17801 raft_consensus.cc:2454] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b79632d31c44d8b62e38b64d84455f in term 0.
I20260620 20:12:38.572618 18013 leader_election.cc:302] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b79632d31c44d8b62e38b64d84455f, e376c5712cf643838c668481f680fd7e; no voters:
I20260620 20:12:38.573025 18160 raft_consensus.cc:2781] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:12:38.573173 18160 raft_consensus.cc:482] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:12:38.573278 18160 raft_consensus.cc:3037] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:38.575577 18160 raft_consensus.cc:504] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } }
I20260620 20:12:38.576138 18160 leader_election.cc:288] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [CANDIDATE]: Term 1 election: Requested vote from peers b7e1dc2fd7dc48938c207b3d3fc8463f (127.4.169.129:42991), e376c5712cf643838c668481f680fd7e (127.4.169.130:36545)
I20260620 20:12:38.576607 17945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af093a7eefac4628a864fc49ba9c3822" candidate_uuid: "13b79632d31c44d8b62e38b64d84455f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e376c5712cf643838c668481f680fd7e"
I20260620 20:12:38.576604 17801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af093a7eefac4628a864fc49ba9c3822" candidate_uuid: "13b79632d31c44d8b62e38b64d84455f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f"
I20260620 20:12:38.576809 17945 raft_consensus.cc:3037] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:38.576869 17801 raft_consensus.cc:3037] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:38.579062 17945 raft_consensus.cc:2454] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b79632d31c44d8b62e38b64d84455f in term 1.
I20260620 20:12:38.579243 17801 raft_consensus.cc:2454] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b79632d31c44d8b62e38b64d84455f in term 1.
I20260620 20:12:38.579577 18013 leader_election.cc:302] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b79632d31c44d8b62e38b64d84455f, e376c5712cf643838c668481f680fd7e; no voters:
I20260620 20:12:38.579912 18160 raft_consensus.cc:2781] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:12:38.580267 18160 raft_consensus.cc:686] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 1 LEADER]: Becoming Leader. State: Replica: 13b79632d31c44d8b62e38b64d84455f, State: Running, Role: LEADER
I20260620 20:12:38.580729 18160 consensus_queue.cc:237] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } }
I20260620 20:12:38.585371 17649 catalog_manager.cc:5495] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f reported cstate change: term changed from 0 to 1, leader changed from <none> to 13b79632d31c44d8b62e38b64d84455f (127.4.169.131). New cstate: current_term: 1 leader_uuid: "13b79632d31c44d8b62e38b64d84455f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b79632d31c44d8b62e38b64d84455f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 33583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 } health_report { overall_health: UNKNOWN } } }
I20260620 20:12:38.646085 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.639822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.4.169.131:33583)
0620 20:12:38.640527 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:38.640531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:38.640553 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:12:38.640641 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:38.640644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:38.640741 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:38.640876 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:12:38.641270 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.642024 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.643387 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.644066 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.644189 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.644673 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:12:38.645373 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:12:38.645509 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:38.645635 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 20:12:38.645720 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":458,"threads_started":1}
I20260620 20:12:38.655125 17945 raft_consensus.cc:1261] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [term 1 FOLLOWER]: Refusing update from remote peer 13b79632d31c44d8b62e38b64d84455f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:12:38.655301 17801 raft_consensus.cc:1261] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [term 1 FOLLOWER]: Refusing update from remote peer 13b79632d31c44d8b62e38b64d84455f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:12:38.656284 18165 consensus_queue.cc:1035] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e376c5712cf643838c668481f680fd7e" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:12:38.656632 18160 consensus_queue.cc:1035] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7e1dc2fd7dc48938c207b3d3fc8463f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:12:38.683815 17648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:12:38.684110 17648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:12:38.687974 17648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f9a9ea8663741928f07626f82afd2c3: Sending DeleteTablet for 3 replicas of tablet af093a7eefac4628a864fc49ba9c3822
I20260620 20:12:38.689209 17781 tablet_service.cc:1514] Processing DeleteTablet for tablet af093a7eefac4628a864fc49ba9c3822 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:12:38 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:41346
I20260620 20:12:38.689256 18059 tablet_service.cc:1514] Processing DeleteTablet for tablet af093a7eefac4628a864fc49ba9c3822 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:12:38 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:39518
I20260620 20:12:38.691666 17925 tablet_service.cc:1514] Processing DeleteTablet for tablet af093a7eefac4628a864fc49ba9c3822 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:12:38 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:45682
I20260620 20:12:38.695546 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 17686
I20260620 20:12:38.703028 18184 tablet_replica.cc:330] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: stopping tablet replica
I20260620 20:12:38.705680 18184 raft_consensus.cc:2229] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:12:38.706307 18184 raft_consensus.cc:2258] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:12:38.708936 18184 ts_tablet_manager.cc:1905] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:12:38.709487 18183 tablet_replica.cc:330] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: stopping tablet replica
I20260620 20:12:38.709960 18183 raft_consensus.cc:2229] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:12:38.710510 18183 raft_consensus.cc:2258] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:12:38.719194 18184 ts_tablet_manager.cc:1918] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:12:38.719313 18183 ts_tablet_manager.cc:1905] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:12:38.719549 18184 log.cc:1196] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/af093a7eefac4628a864fc49ba9c3822
I20260620 20:12:38.720233 18184 ts_tablet_manager.cc:1939] T af093a7eefac4628a864fc49ba9c3822 P e376c5712cf643838c668481f680fd7e: Deleting consensus metadata
I20260620 20:12:38.720404 18182 tablet_replica.cc:330] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: stopping tablet replica
I20260620 20:12:38.722379 17634 catalog_manager.cc:4841] TS e376c5712cf643838c668481f680fd7e (127.4.169.130:36545): tablet af093a7eefac4628a864fc49ba9c3822 (table test-table [id=8e27bb49299d4d09806ce54ebcf4de99]) successfully deleted
I20260620 20:12:38.723001 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.696588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.4.169.129:42991)
0620 20:12:38.697479 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:38.697489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:38.700949 (+ 3460us) server_negotiation.cc:407] Connection header received
0620 20:12:38.701159 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:38.701168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:38.701294 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:38.701407 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:38.705233 (+ 3826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.706336 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.710922 (+ 4586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.711557 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.711698 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.715319 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:38.715334 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:38.715353 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:38.715405 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:38.719511 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:38.720531 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:38.720543 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:38.720548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:38.720764 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:38.721173 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:38.721179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:38.721182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:38.721648 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:38.721965 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:38.722284 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 20:12:38.722583 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648}
I20260620 20:12:38.723423 18182 raft_consensus.cc:2229] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:12:38.723875 17841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:38.724228 18182 raft_consensus.cc:2258] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:12:38.726958 18182 ts_tablet_manager.cc:1905] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:12:38.733108 18182 ts_tablet_manager.cc:1918] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:12:38.733343 18182 log.cc:1196] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/af093a7eefac4628a864fc49ba9c3822
I20260620 20:12:38.733857 18182 ts_tablet_manager.cc:1939] T af093a7eefac4628a864fc49ba9c3822 P 13b79632d31c44d8b62e38b64d84455f: Deleting consensus metadata
I20260620 20:12:38.735921 17634 catalog_manager.cc:4841] TS 13b79632d31c44d8b62e38b64d84455f (127.4.169.131:33583): tablet af093a7eefac4628a864fc49ba9c3822 (table test-table [id=8e27bb49299d4d09806ce54ebcf4de99]) successfully deleted
I20260620 20:12:38.871800 18183 ts_tablet_manager.cc:1918] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:12:38.872022 18183 log.cc:1196] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/af093a7eefac4628a864fc49ba9c3822
I20260620 20:12:38.872521 18183 ts_tablet_manager.cc:1939] T af093a7eefac4628a864fc49ba9c3822 P b7e1dc2fd7dc48938c207b3d3fc8463f: Deleting consensus metadata
I20260620 20:12:38.874300 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 17686
I20260620 20:12:38.876142 17636 catalog_manager.cc:4841] TS b7e1dc2fd7dc48938c207b3d3fc8463f (127.4.169.129:42991): tablet af093a7eefac4628a864fc49ba9c3822 (table test-table [id=8e27bb49299d4d09806ce54ebcf4de99]) successfully deleted
I20260620 20:12:38.891976 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 17850
I20260620 20:12:38.907827 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:38.892752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.4.169.130:36545)
0620 20:12:38.892985 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:38.892992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:38.893018 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:12:38.893230 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:38.893234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:38.893302 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:38.893397 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:38.895254 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.896011 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.897824 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:38.898270 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:38.898367 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:38.901593 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:38.901607 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:38.901622 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:38.901653 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:38.904636 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:38.905516 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:38.905524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:38.905526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:38.905649 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:38.906066 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:38.906071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:38.906073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:38.906725 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:38.907015 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:38.907326 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 20:12:38.907496 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:12:38.908644 17986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:38.999490 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 17850
I20260620 20:12:39.020920 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 17992
I20260620 20:12:39.039577 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:39.021708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.4.169.131:33583)
0620 20:12:39.021903 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:39.021910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:39.021936 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:12:39.022169 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:39.022174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:39.022236 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:39.022350 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:39.024211 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:39.025153 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:39.027088 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:39.027543 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:39.027645 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:39.031477 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:39.031494 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:39.031546 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:39.031636 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:39.036362 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:39.037305 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:39.037312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:39.037314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:39.037454 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:39.037926 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:39.037930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:39.037932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:39.038318 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:39.038665 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:39.039027 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 20:12:39.039212 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:12:39.040347 18119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:39.161227 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 17992
I20260620 20:12:39.176113 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 17616
I20260620 20:12:39.177287 17679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:39.289554 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 17616
I20260620 20:12:39.311583 4774 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:12:39.647089 4774 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:12:39.647274 4774 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:12:39.993127 4774 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:12:39.993 UTC [16668] LOG: received smart shutdown request
2026-06-20 20:12:39.996 UTC [16668] LOG: background worker "logical replication launcher" (PID 16677) exited with exit code 1
2026-06-20 20:12:39.996 UTC [16672] LOG: shutting down
2026-06-20 20:12:40.140 UTC [16668] LOG: database system is shut down
2026-06-20T20:12:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54101 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:12:40 dist-test-slave-4bwq krb5kdc[18198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:12:40 dist-test-slave-4bwq krb5kdc[18198](info): set up 2 sockets
Jun 20 20:12:40 dist-test-slave-4bwq krb5kdc[18198](info): commencing operation
krb5kdc: starting...
W20260620 20:12:41.374966 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:12:41 dist-test-slave-4bwq krb5kdc[18198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:12:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:12:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:12:41.605899 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:39685
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:39685,127.4.169.189:35839,127.4.169.188:37087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:41.946790 18211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:41.947163 18211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:41.947341 18211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:41.954595 18211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:12:41.954753 18211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:41.954865 18211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:41.954977 18211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:12:41.955044 18211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:12:41.964896 18211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:39685,127.4.169.189:35839,127.4.169.188:37087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:39685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:41.966316 18211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:41.967700 18211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:41.979866 18217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:41.979957 18216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:41.980022 18219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:41.980517 18211 server_base.cc:1034] running on GCE node
I20260620 20:12:41.981326 18211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:41.982798 18211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:41.984033 18211 hybrid_clock.cc:648] HybridClock initialized: now 1781986361984000 us; error 48 us; skew 500 ppm
I20260620 20:12:41.984385 18211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:41 dist-test-slave-4bwq krb5kdc[18198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986361, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:41.987619 18211 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:12:41.989315 18211 webserver.cc:460] Webserver started at http://127.4.169.190:43785/ using document root <none> and password file <none>
I20260620 20:12:41.990074 18211 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:41.990222 18211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:41.990592 18211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:12:41.993602 18211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f"
format_stamp: "Formatted at 2026-06-20 20:12:41 on dist-test-slave-4bwq"
server_key: "1cdde11c1d43c1c7fbcb5299ac236e27"
server_key_iv: "5715a20a3c88a9ce9115ca6def99ffb9"
server_key_version: "encryptionkey@0"
I20260620 20:12:41.994478 18211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f"
format_stamp: "Formatted at 2026-06-20 20:12:41 on dist-test-slave-4bwq"
server_key: "1cdde11c1d43c1c7fbcb5299ac236e27"
server_key_iv: "5715a20a3c88a9ce9115ca6def99ffb9"
server_key_version: "encryptionkey@0"
I20260620 20:12:41.999657 18211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:12:42.003291 18226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:42.004840 18211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:12:42.005060 18211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f"
format_stamp: "Formatted at 2026-06-20 20:12:41 on dist-test-slave-4bwq"
server_key: "1cdde11c1d43c1c7fbcb5299ac236e27"
server_key_iv: "5715a20a3c88a9ce9115ca6def99ffb9"
server_key_version: "encryptionkey@0"
I20260620 20:12:42.005295 18211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:42.027441 18211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:42.031790 18211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:42.032085 18211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:42.051379 18211 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:39685
I20260620 20:12:42.051419 18277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:39685 every 8 connection(s)
I20260620 20:12:42.052778 18211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:12:42.056593 18278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:42.060739 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 18211
I20260620 20:12:42.060992 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:12:42.061817 4774 external_mini_cluster.cc:1428] Setting key 36f7cb363769ebedd1e178b38609440d
I20260620 20:12:42.064872 18278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:12:42 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986361, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.169.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.189@KRBTEST.COM" created.
I20260620 20:12:42.091929 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.069160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:39685 (local address 127.0.0.1:42714)
0620 20:12:42.070019 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.070068 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:42.073126 (+ 3058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:42.074064 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:42.074074 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:42.074995 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:42.075528 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:42.075556 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.077216 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.077224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:42.079160 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:42.079178 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.080189 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.080199 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:42.080452 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:42.081234 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:42.081279 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:42.084095 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:42.087494 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:42.087507 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:42.087520 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:42.087938 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:42.088417 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:42.088424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:42.088428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:42.088585 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:42.089397 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:42.089405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:42.089610 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 20:12:42.089860 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 20:12:42.090490 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":149,"thread_start_us":380,"threads_started":1}
I20260620 20:12:42.091969 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.070563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.4.169.190:39685)
0620 20:12:42.071304 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:42.071310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:42.071355 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:12:42.073537 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:42.073544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:42.073787 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:42.073966 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:42.075756 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.077026 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.079317 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.079919 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.080056 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:42.084349 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:42.084357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:42.084373 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:42.084405 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:42.087356 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:42.088080 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:42.088093 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:42.088096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:42.088297 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:42.088701 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:42.088706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:42.088709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:42.089252 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:42.089469 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:42.089988 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 20:12:42.090333 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":540,"thread_start_us":402,"threads_started":1}
I20260620 20:12:42.095327 18278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:12:42.096498 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.096085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56858)
0620 20:12:42.096262 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.096381 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 20:12:42.096839 18229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
I20260620 20:12:42.098553 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.098135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56868)
0620 20:12:42.098294 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.098362 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:12:42.098771 18278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:35839: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab.
Entry for principal kudu/127.4.169.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab.
I20260620 20:12:42.115208 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.189:35839
--webserver_interface=127.4.169.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab
--principal=kudu/127.4.169.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:39685,127.4.169.189:35839,127.4.169.188:37087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:12:42.148095 18278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } attempt: 1
I20260620 20:12:42.149349 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.148790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56872)
0620 20:12:42.149021 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.149180 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 20:12:42.150450 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.149997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56876)
0620 20:12:42.150157 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.150271 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:12:42.150669 18278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:35839: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
I20260620 20:12:42.218927 18278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } attempt: 2
I20260620 20:12:42.220219 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.219661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56878)
0620 20:12:42.219926 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.220048 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 20:12:42.221242 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.220832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56884)
0620 20:12:42.221008 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.221093 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:12:42.221469 18278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:35839: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
I20260620 20:12:42.312757 18278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } attempt: 3
I20260620 20:12:42.314021 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.313490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56896)
0620 20:12:42.313721 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.313854 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 20:12:42.315169 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.314748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56904)
0620 20:12:42.314918 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.315005 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 20:12:42.315371 18278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:35839: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
W20260620 20:12:42.453321 18285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:42.453636 18285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:42.453804 18285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 20:12:42.458628 18278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } attempt: 4
I20260620 20:12:42.459915 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.459293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56916)
0620 20:12:42.459640 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.459749 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260620 20:12:42.460961 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.460551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56918)
0620 20:12:42.460709 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.460808 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:12:42.461231 18278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:35839: Network error: Client connection negotiation failed: client connection to 127.4.169.189:35839: connect: Connection refused (error 111)
W20260620 20:12:42.462276 18285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:12:42.462450 18285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:42.462563 18285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:42.462662 18285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:12:42.462749 18285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:12:42.473253 18285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:39685,127.4.169.189:35839,127.4.169.188:37087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab
--principal=kudu/127.4.169.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.189:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.169.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:42.475023 18285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:42.476454 18285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:42.488517 18293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:42.488605 18291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:42.488560 18290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:42.489794 18285 server_base.cc:1034] running on GCE node
I20260620 20:12:42.490665 18285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:42.492122 18285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:42.493352 18285 hybrid_clock.cc:648] HybridClock initialized: now 1781986362493313 us; error 53 us; skew 500 ppm
I20260620 20:12:42.493698 18285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:42 dist-test-slave-4bwq krb5kdc[18198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:42.496922 18285 init.cc:377] Logged in from keytab as kudu/127.4.169.189@KRBTEST.COM (short username kudu)
I20260620 20:12:42.498584 18285 webserver.cc:460] Webserver started at http://127.4.169.189:44603/ using document root <none> and password file <none>
I20260620 20:12:42.499281 18285 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:42.499410 18285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:42.499725 18285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:12:42.502771 18285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/data/instance:
uuid: "4a0c1377773e425bb07114baf26a4767"
format_stamp: "Formatted at 2026-06-20 20:12:42 on dist-test-slave-4bwq"
server_key: "c5e393b6aa2e81cd2c35eca21656143e"
server_key_iv: "0d46209f03e17b3999fd4a327147c18e"
server_key_version: "encryptionkey@0"
I20260620 20:12:42.503599 18285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/wal/instance:
uuid: "4a0c1377773e425bb07114baf26a4767"
format_stamp: "Formatted at 2026-06-20 20:12:42 on dist-test-slave-4bwq"
server_key: "c5e393b6aa2e81cd2c35eca21656143e"
server_key_iv: "0d46209f03e17b3999fd4a327147c18e"
server_key_version: "encryptionkey@0"
I20260620 20:12:42.509020 18285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:12:42.512580 18300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:42.513962 18285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:12:42.514153 18285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/wal
uuid: "4a0c1377773e425bb07114baf26a4767"
format_stamp: "Formatted at 2026-06-20 20:12:42 on dist-test-slave-4bwq"
server_key: "c5e393b6aa2e81cd2c35eca21656143e"
server_key_iv: "0d46209f03e17b3999fd4a327147c18e"
server_key_version: "encryptionkey@0"
I20260620 20:12:42.514365 18285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:42.539177 18285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:42.543713 18285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:42.544027 18285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:42.563476 18285 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.189:35839
I20260620 20:12:42.563511 18351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.189:35839 every 8 connection(s)
I20260620 20:12:42.564881 18285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
I20260620 20:12:42.568593 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 18285
I20260620 20:12:42.568629 18352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:42.568799 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-1/wal/instance
I20260620 20:12:42.569260 4774 external_mini_cluster.cc:1428] Setting key efc9b99c8004abe7061fc6883c7c3e14
I20260620 20:12:42.575316 18352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.169.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.188@KRBTEST.COM" created.
Jun 20 20:12:42 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:42.599928 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.578226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.4.169.190:39685)
0620 20:12:42.578489 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:42.578497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:42.579881 (+ 1384us) server_negotiation.cc:407] Connection header received
0620 20:12:42.581795 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:42.581805 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:42.581902 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:42.582027 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:42.583650 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.584800 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.586534 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.587249 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.587410 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:42.592586 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:42.592596 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:42.592605 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:42.592656 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:42.596450 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:42.597241 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:42.597249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:42.597255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:42.597409 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:42.597822 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:42.597830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:42.597836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:42.598311 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:42.598595 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:42.599331 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 20:12:42.599579 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:12:42.600952 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.578819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:39685 (local address 127.0.0.1:42720)
0620 20:12:42.579668 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.579728 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:42.581511 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:42.582255 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:42.582264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:42.582860 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:42.583450 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:42.583467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.584995 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.585003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:42.586308 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:42.586321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.587476 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.587482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:42.587679 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:42.588355 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:42.588382 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:42.592306 (+ 3924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:42.596621 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:42.596628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:42.596643 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:42.597099 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:42.597520 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:42.597523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:42.597526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:42.597702 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:42.598534 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:42.598540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:42.598915 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 20:12:42.599213 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 20:12:42.599791 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":377,"threads_started":1}
I20260620 20:12:42.603044 18352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:12:42 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:12:42.619001 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.603981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56930)
0620 20:12:42.604185 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.604204 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:42.604328 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:42.605465 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:42.605469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:42.605941 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:42.606081 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:42.606090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.607577 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.607584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:42.608416 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:42.608428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.609796 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.609825 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:42.610051 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:42.610761 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:42.610779 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:42.612729 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:42.616300 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:42.616304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:42.616307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:42.616689 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:42.617141 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:42.617144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:42.617147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:42.617297 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:42.618083 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:42.618087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:42.618206 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 20:12:42.618345 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:12:42.618620 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:12:42.619369 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.604208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.4.169.189:35839)
0620 20:12:42.604939 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:42.604945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:42.604970 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:12:42.605016 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:42.605021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:42.605212 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:42.605480 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:42.606234 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.607439 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.608594 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.609624 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.609774 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:42.612968 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:42.612988 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:42.613008 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:42.613053 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:42.616144 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:42.616851 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:42.616861 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:42.616864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:42.617012 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:42.617478 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:42.617481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:42.617483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:42.617935 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:42.618148 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:42.618542 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 20:12:42.618822 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":405,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab.
Entry for principal kudu/127.4.169.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab.
I20260620 20:12:42.621337 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.188:37087
--webserver_interface=127.4.169.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab
--principal=kudu/127.4.169.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:39685,127.4.169.189:35839,127.4.169.188:37087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:12:42.621730 18352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:12:42.622931 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.622316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34398)
0620 20:12:42.622550 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.622724 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 20:12:42.623201 18304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
I20260620 20:12:42.624934 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.624466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34414)
0620 20:12:42.624675 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.624783 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 20:12:42.625178 18352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:37087: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
I20260620 20:12:42.674540 18352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } attempt: 1
I20260620 20:12:42.675884 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.675420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34416)
0620 20:12:42.675613 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.675720 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 20:12:42.676852 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.676511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34426)
0620 20:12:42.676675 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.676740 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:12:42.677063 18352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:37087: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
I20260620 20:12:42.745275 18352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } attempt: 2
I20260620 20:12:42.746546 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.746013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34438)
0620 20:12:42.746205 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.746295 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 20:12:42.747545 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.747204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34452)
0620 20:12:42.747352 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.747436 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:12:42.747741 18352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:37087: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
I20260620 20:12:42.760493 18278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } attempt: 5
Jun 20 20:12:42 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986361, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:12:42.773903 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.761320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.4.169.189:35839)
0620 20:12:42.761544 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:42.761551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:42.761651 (+ 100us) server_negotiation.cc:407] Connection header received
0620 20:12:42.761860 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:42.761865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:42.761958 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:42.762069 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:42.762917 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.763625 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.764779 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.765234 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.765378 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:42.768533 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:42.768538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:42.768541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:42.768575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:42.771286 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:42.771969 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:42.771976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:42.771980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:42.772118 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:42.772488 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:42.772494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:42.772497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:42.772783 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:42.772943 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:42.773334 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 20:12:42.773562 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260620 20:12:42.773914 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.761292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56946)
0620 20:12:42.761543 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.761565 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:42.761728 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:42.762104 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:42.762109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:42.762574 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:42.762765 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:42.762774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.763760 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.763767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:42.764632 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:42.764644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:42.765386 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:42.765391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:42.765578 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:42.766212 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:42.766231 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:42.768330 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:42.771449 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:42.771455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:42.771459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:42.771842 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:42.772235 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:42.772239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:42.772243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:42.772382 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:42.772920 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:42.772925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:42.773087 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 20:12:42.773234 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:12:42.773536 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 20:12:42.775225 18278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:12:42.776460 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.776011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34454)
0620 20:12:42.776234 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.776312 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 20:12:42.777467 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.777141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34456)
0620 20:12:42.777297 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.777373 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:12:42.777673 18278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:37087: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
I20260620 20:12:42.828904 18278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } attempt: 1
I20260620 20:12:42.830117 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.829573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34470)
0620 20:12:42.829817 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.829958 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 20:12:42.831223 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.830893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34478)
0620 20:12:42.831041 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.831115 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:12:42.831440 18278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:37087: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
I20260620 20:12:42.838949 18352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } attempt: 3
I20260620 20:12:42.840153 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.839582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34490)
0620 20:12:42.839809 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.839923 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:12:42.841419 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.840892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34502)
0620 20:12:42.841114 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.841232 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260620 20:12:42.841696 18352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:37087: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
I20260620 20:12:42.899657 18278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } attempt: 2
I20260620 20:12:42.900768 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.900273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34516)
0620 20:12:42.900517 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.900604 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 20:12:42.901839 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.901510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34522)
0620 20:12:42.901663 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.901736 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:12:42.902051 18278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:37087: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
W20260620 20:12:42.953429 18359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:42.953791 18359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:42.953966 18359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:42.961326 18359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:12:42.961511 18359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:42.961621 18359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:42.961726 18359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:12:42.961804 18359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:12:42.972062 18359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:39685,127.4.169.189:35839,127.4.169.188:37087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab
--principal=kudu/127.4.169.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.188:37087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.169.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:42.973490 18359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:42.974843 18359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:12:42.984927 18352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } attempt: 4
I20260620 20:12:42.986428 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.985747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34538)
0620 20:12:42.986086 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.986198 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260620 20:12:42.986821 18365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:42.986820 18367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:42.987396 18364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:42.987885 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:42.987521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34550)
0620 20:12:42.987700 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:42.987773 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 20:12:42.988109 18352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:37087: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
I20260620 20:12:42.988301 18359 server_base.cc:1034] running on GCE node
I20260620 20:12:42.989105 18359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:42.990686 18359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:42.991942 18359 hybrid_clock.cc:648] HybridClock initialized: now 1781986362991869 us; error 66 us; skew 500 ppm
I20260620 20:12:42.992352 18359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:42 dist-test-slave-4bwq krb5kdc[18198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:42.995728 18359 init.cc:377] Logged in from keytab as kudu/127.4.169.188@KRBTEST.COM (short username kudu)
I20260620 20:12:42.997212 18359 webserver.cc:460] Webserver started at http://127.4.169.188:45629/ using document root <none> and password file <none>
I20260620 20:12:42.997941 18359 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:42.998088 18359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:42.998464 18359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:12:43.001592 18359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/data/instance:
uuid: "5753549445aa461d80a314d89a2fc823"
format_stamp: "Formatted at 2026-06-20 20:12:42 on dist-test-slave-4bwq"
server_key: "0afca72f1f5c2f65ddd59b7defd2d116"
server_key_iv: "ac2c2d315cc4adf2ab4fb27e85d93b2e"
server_key_version: "encryptionkey@0"
I20260620 20:12:43.002501 18359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/wal/instance:
uuid: "5753549445aa461d80a314d89a2fc823"
format_stamp: "Formatted at 2026-06-20 20:12:42 on dist-test-slave-4bwq"
server_key: "0afca72f1f5c2f65ddd59b7defd2d116"
server_key_iv: "ac2c2d315cc4adf2ab4fb27e85d93b2e"
server_key_version: "encryptionkey@0"
I20260620 20:12:43.007938 18359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:12:43.008246 18278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } attempt: 3
I20260620 20:12:43.009514 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.008924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34552)
0620 20:12:43.009187 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:43.009298 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 20:12:43.010692 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.010224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34564)
0620 20:12:43.010469 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:43.010560 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 20:12:43.010914 18278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:37087: Network error: Client connection negotiation failed: client connection to 127.4.169.188:37087: connect: Connection refused (error 111)
I20260620 20:12:43.012307 18374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:43.013764 18359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:12:43.013940 18359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/wal
uuid: "5753549445aa461d80a314d89a2fc823"
format_stamp: "Formatted at 2026-06-20 20:12:42 on dist-test-slave-4bwq"
server_key: "0afca72f1f5c2f65ddd59b7defd2d116"
server_key_iv: "ac2c2d315cc4adf2ab4fb27e85d93b2e"
server_key_version: "encryptionkey@0"
I20260620 20:12:43.014154 18359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:43.049446 18359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:43.054463 18359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:43.054787 18359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:43.074699 18359 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.188:37087
I20260620 20:12:43.074719 18425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.188:37087 every 8 connection(s)
I20260620 20:12:43.076148 18359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
I20260620 20:12:43.077244 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 18359
I20260620 20:12:43.077440 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986152907614-4774-0/minicluster-data/master-2/wal/instance
I20260620 20:12:43.077952 4774 external_mini_cluster.cc:1428] Setting key 20d68d053576054ff7ffb157c5f8fb3c
I20260620 20:12:43.080857 18426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:43.090472 18426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:12:43 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
Jun 20 20:12:43 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:43.104442 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.080099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.4.169.188:37087)
0620 20:12:43.080944 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.080960 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.081022 (+ 62us) server_negotiation.cc:407] Connection header received
0620 20:12:43.083374 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.083406 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.083936 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.084427 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.085833 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.088045 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.089669 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.090823 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.091030 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.095869 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.095877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.095886 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.095934 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.099591 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.100520 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.100530 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.100539 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.100768 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.101224 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.101228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.101231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.101805 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.102049 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.102637 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.103147 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":364,"threads_started":1}
I20260620 20:12:43.110181 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.093519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.4.169.190:39685)
0620 20:12:43.093760 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.093768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.094206 (+ 438us) server_negotiation.cc:407] Connection header received
0620 20:12:43.094487 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.094493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.094559 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.094677 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.096193 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.097271 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.098978 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.099722 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.099839 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.103710 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.103717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.103720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.103755 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.106950 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.107795 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.107801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.107805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.107996 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.108452 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.108459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.108463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.108884 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.109055 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.109641 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.109832 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":98}
I20260620 20:12:43.110440 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.093307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:39685 (local address 127.0.0.1:42726)
0620 20:12:43.094061 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:43.094099 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:43.094307 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:43.094772 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:43.094782 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:43.095637 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:43.096031 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.096041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.097465 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.097469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.098814 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.098829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.099917 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.099926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.100142 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.100864 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:43.100898 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:43.103516 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:43.107084 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.107096 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.107102 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.107642 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.108180 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.108185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.108189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.108341 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.109082 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:43.109091 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:43.109342 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 20:12:43.109564 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 20:12:43.109934 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":327,"threads_started":1}
I20260620 20:12:43.112066 18426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:12:43 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
Jun 20 20:12:43 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:12:43.123852 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.107805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.4.169.190:39685)
0620 20:12:43.108417 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.108422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.108443 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:12:43.108483 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.108487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.108558 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.108669 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.110335 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.111473 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.113089 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.113530 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.113668 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.117870 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.117883 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.117887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.117918 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.121041 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.121749 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.121756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.121760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.121891 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.122278 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.122283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.122286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.122666 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.122943 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.123245 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.123469 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":472,"thread_start_us":270,"threads_started":1}
I20260620 20:12:43.125528 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.112841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:35839 (local address 127.0.0.1:56950)
0620 20:12:43.113015 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:43.113036 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:43.113129 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:43.113426 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:43.113430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:43.113784 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:43.113935 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.113943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.114984 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.114993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.115849 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.115868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.116563 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.116569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.116821 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.117413 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:43.117429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:43.119963 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:43.123045 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.123050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.123054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.123428 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.123870 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.123875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.123879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.124080 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.124565 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:43.124570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:43.124734 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 20:12:43.124881 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:12:43.125166 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 20:12:43.125584 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.112861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.4.169.189:35839)
0620 20:12:43.113131 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.113138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.113157 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:12:43.113221 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.113226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.113296 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.113400 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.114092 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.114851 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.116004 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.116414 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.116548 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.120175 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.120180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.120183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.120216 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.122904 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.123555 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.123560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.123568 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.123746 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.124189 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.124195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.124198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.124451 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.124611 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.124960 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.125166 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260620 20:12:43.127111 18426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:12:43 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
Jun 20 20:12:43 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:12:43.140130 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.127940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.4.169.188:37087)
0620 20:12:43.128114 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.128119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.128292 (+ 173us) server_negotiation.cc:407] Connection header received
0620 20:12:43.128472 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.128476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.128538 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.128629 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.129264 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.129937 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.131096 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.131755 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.131940 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.134884 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.134890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.134894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.134930 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.137581 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.138183 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.138190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.138194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.138312 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.138685 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.138693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.138696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.138960 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.139107 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.139427 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.139707 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":61}
I20260620 20:12:43.140246 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.127969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34594)
0620 20:12:43.128213 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:43.128226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:43.128345 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:43.128644 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:43.128647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:43.129015 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:43.129138 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.129146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.130055 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.130063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.130965 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.130976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.131930 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.131937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.132125 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.132653 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:43.132670 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:43.134727 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:43.137716 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.137721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.137725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.138069 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.138448 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.138453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.138457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.138584 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.139070 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:43.139074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:43.139213 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 20:12:43.139342 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 20:12:43.139846 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":172}
I20260620 20:12:43.140973 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.125599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.4.169.189:35839)
0620 20:12:43.125878 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.125883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.125904 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:12:43.126151 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.126156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.126246 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.126326 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.128180 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.128923 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.130178 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.130664 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.130794 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.135589 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.135594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.135597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.135628 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.138212 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.138985 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.138992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.138996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.139142 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.139548 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.139552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.139554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.139807 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.140083 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.140461 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.140704 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":398,"threads_started":1}
I20260620 20:12:43.145920 18426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823: Bootstrap starting.
I20260620 20:12:43.146848 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:12:43.149415 18426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:43.150743 18426 log.cc:824] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:43.154470 18426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823: No bootstrap required, opened a new log
I20260620 20:12:43.170136 18426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } }
I20260620 20:12:43.170858 18426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:43.171078 18426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5753549445aa461d80a314d89a2fc823, State: Initialized, Role: FOLLOWER
I20260620 20:12:43.172065 18426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } }
I20260620 20:12:43.179323 18441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.182672 18441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:12:43.182551 18426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:12:43.183454 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.152161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.4.169.190:39685)
0620 20:12:43.152361 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.152367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.163245 (+ 10878us) server_negotiation.cc:407] Connection header received
0620 20:12:43.163480 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.163489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.163607 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.163743 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.166434 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.167555 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.169918 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.170617 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.170745 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.175431 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.175439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.175445 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.175495 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.179728 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.180891 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.180897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.180901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.181058 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.181527 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.181532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.181535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.181945 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.182169 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.182829 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.183085 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:12:43.183554 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.151826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.4.169.188:37087)
0620 20:12:43.152031 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.152037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.164357 (+ 12320us) server_negotiation.cc:407] Connection header received
0620 20:12:43.164902 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.164908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.165002 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.165083 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.167476 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.168691 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.170825 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.171527 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.171648 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.175572 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.175580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.175587 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.175630 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.179854 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.181003 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.181009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.181012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.181179 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.181672 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.181677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.181681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.182100 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.182335 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.182951 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.183178 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:12:43.183727 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.151766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.4.169.189:35839)
0620 20:12:43.151964 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.151970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.164029 (+ 12059us) server_negotiation.cc:407] Connection header received
0620 20:12:43.164229 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.164234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.164313 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.164417 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.166548 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.167657 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.170020 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.170657 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.170765 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.176484 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.176493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.176499 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.176550 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.180270 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.181206 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.181211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.181215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.181389 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.181863 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.181868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.181871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.182244 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.182489 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.183085 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.183319 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:12:43.188127 18278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } attempt: 4
W20260620 20:12:43.196468 18452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:12:43.196600 18452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:12:43.193531 18359 master_runner.cc:186] Error getting master registration for 127.4.169.190:39685: OK, instance_id { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" instance_seqno: 1781986362045595 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Jun 20 20:12:43 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986361, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
I20260620 20:12:43.203094 18359 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:12:43.210027 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.188673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34612)
0620 20:12:43.188843 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:43.188861 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:43.188958 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:43.189336 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:43.189339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:43.189670 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:43.189831 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.189839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.191565 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.191571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.192947 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.192965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.199829 (+ 6864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.199835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.200046 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.200773 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:43.200812 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:43.203678 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:43.207067 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.207076 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.207081 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.207658 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.208128 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.208134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.208137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.208315 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.208936 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:43.208941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:43.209140 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 20:12:43.209301 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 20:12:43.209611 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:12:43.210152 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.188770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.4.169.188:37087)
0620 20:12:43.188982 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.188990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.189014 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:12:43.189078 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.189085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.189175 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.189304 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.189980 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.191026 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.198545 (+ 7519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.199429 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.199582 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.203886 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.203891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.203895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.203925 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.206920 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.207778 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.207783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.207787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.207969 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.208451 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.208454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.208456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.208799 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.208906 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.209422 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.209738 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:12:43.214949 18278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f: Bootstrap starting.
I20260620 20:12:43.219347 18278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:43.220670 18278 log.cc:824] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:43.223472 18278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f: No bootstrap required, opened a new log
I20260620 20:12:43.231245 18278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } }
I20260620 20:12:43.231662 18278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:43.231815 18278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d1d3ee3da941f5a8f8ec9800ebb78f, State: Initialized, Role: FOLLOWER
I20260620 20:12:43.232445 18278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } }
I20260620 20:12:43.233377 18455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.233775 18455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:12:43.234661 18278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:12:43.246628 18466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:12:43.246883 18466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:12:43.244917 18211 master_runner.cc:186] Error getting master registration for 127.4.169.189:35839: OK, instance_id { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" instance_seqno: 1781986362557786 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:12:43.255308 18211 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:12:43.284948 18232 tablet.cc:2369] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:12:43.287340 18352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } attempt: 5
Jun 20 20:12:43 dist-test-slave-4bwq krb5kdc[18198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986362, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
I20260620 20:12:43.305446 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.288136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.4.169.188:37087)
0620 20:12:43.288344 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:43.288350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:43.288427 (+ 77us) server_negotiation.cc:407] Connection header received
0620 20:12:43.288627 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:43.288633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:43.288716 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:43.288845 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:43.289848 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.291026 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.293054 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.293799 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.293927 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.298245 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:43.298253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:43.298258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:43.298300 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:43.302196 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.303141 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.303147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.303151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.303319 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:43.303732 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:43.303737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:43.303740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:43.304121 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:43.304303 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:43.304775 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 20:12:43.305037 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 20:12:43.307231 18380 tablet.cc:2369] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:12:43.307940 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:43.288046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:37087 (local address 127.0.0.1:34626)
0620 20:12:43.288329 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:43.288359 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:43.288486 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:43.288880 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:43.288885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:43.289413 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:43.289626 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.289639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.291459 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.291465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.292901 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:43.292917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:43.294145 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:43.294154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:43.294446 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:43.295167 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:43.295191 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:43.298049 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:43.302371 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.302379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.302408 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.303003 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.303444 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:43.303450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:43.303454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:43.303615 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:43.304259 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:43.304264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:43.304488 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 20:12:43.304658 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 20:12:43.304956 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 20:12:43.314298 18352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767: Bootstrap starting.
I20260620 20:12:43.318609 18352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:43.319955 18352 log.cc:824] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:43.320032 18455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:12:43.320374 18455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } }
I20260620 20:12:43.322027 18455 leader_election.cc:288] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a0c1377773e425bb07114baf26a4767 (127.4.169.189:35839), 5753549445aa461d80a314d89a2fc823 (127.4.169.188:37087)
I20260620 20:12:43.323369 18401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5753549445aa461d80a314d89a2fc823" is_pre_election: true
I20260620 20:12:43.323989 18401 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d1d3ee3da941f5a8f8ec9800ebb78f in term 0.
I20260620 20:12:43.324823 18230 leader_election.cc:302] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5753549445aa461d80a314d89a2fc823, 76d1d3ee3da941f5a8f8ec9800ebb78f; no voters:
I20260620 20:12:43.325965 18455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:12:43.326113 18455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:12:43.326206 18455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:43.329679 18352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767: No bootstrap required, opened a new log
I20260620 20:12:43.334244 18455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } }
I20260620 20:12:43.338792 18401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5753549445aa461d80a314d89a2fc823"
I20260620 20:12:43.339052 18401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:43.341234 18352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } }
I20260620 20:12:43.341638 18352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:43.341744 18352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a0c1377773e425bb07114baf26a4767, State: Initialized, Role: FOLLOWER
I20260620 20:12:43.342561 18352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } }
I20260620 20:12:43.342829 18401 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d1d3ee3da941f5a8f8ec9800ebb78f in term 1.
I20260620 20:12:43.343669 18230 leader_election.cc:302] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5753549445aa461d80a314d89a2fc823, 76d1d3ee3da941f5a8f8ec9800ebb78f; no voters:
I20260620 20:12:43.347816 18455 leader_election.cc:288] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [CANDIDATE]: Term 1 election: Requested vote from peers 4a0c1377773e425bb07114baf26a4767 (127.4.169.189:35839), 5753549445aa461d80a314d89a2fc823 (127.4.169.188:37087)
I20260620 20:12:43.349570 18455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:12:43.349644 18469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.350095 18469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:12:43.351603 18327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a0c1377773e425bb07114baf26a4767" is_pre_election: true
I20260620 20:12:43.352205 18327 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d1d3ee3da941f5a8f8ec9800ebb78f in term 0.
I20260620 20:12:43.352934 18455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [term 1 LEADER]: Becoming Leader. State: Replica: 76d1d3ee3da941f5a8f8ec9800ebb78f, State: Running, Role: LEADER
I20260620 20:12:43.353209 18326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a0c1377773e425bb07114baf26a4767"
I20260620 20:12:43.353452 18326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:43.353546 18455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } }
I20260620 20:12:43.351575 18352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:12:43.356981 18326 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d1d3ee3da941f5a8f8ec9800ebb78f in term 1.
I20260620 20:12:43.360756 18470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d1d3ee3da941f5a8f8ec9800ebb78f. Latest consensus state: current_term: 1 leader_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.361214 18470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:43.362133 18474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:12:43.364430 18474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:12:43.382275 18401 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [term 1 FOLLOWER]: Refusing update from remote peer 76d1d3ee3da941f5a8f8ec9800ebb78f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:12:43.382323 18326 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [term 1 FOLLOWER]: Refusing update from remote peer 76d1d3ee3da941f5a8f8ec9800ebb78f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:12:43.383338 18470 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:12:43.384032 18455 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:12:43.409761 18441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d1d3ee3da941f5a8f8ec9800ebb78f. Latest consensus state: current_term: 1 leader_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.410121 18441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:12:43.411830 18469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d1d3ee3da941f5a8f8ec9800ebb78f. Latest consensus state: current_term: 1 leader_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.412206 18469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:12:43.412977 18475 mvcc.cc:205] Tried to move back new op lower bound from 7299016144400986112 to 7299016144307109888. Current Snapshot: MvccSnapshot[applied={T|T < 7299016144400986112}]
I20260620 20:12:43.415045 18477 mvcc.cc:205] Tried to move back new op lower bound from 7299016144400986112 to 7299016144307109888. Current Snapshot: MvccSnapshot[applied={T|T < 7299016144400986112}]
I20260620 20:12:43.418434 18478 mvcc.cc:205] Tried to move back new op lower bound from 7299016144400986112 to 7299016144307109888. Current Snapshot: MvccSnapshot[applied={T|T < 7299016144400986112}]
I20260620 20:12:43.422945 18469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.423214 18469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:12:43.434183 18441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.434511 18441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:12:43.437130 18455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.437464 18455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:43.437733 18455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d1d3ee3da941f5a8f8ec9800ebb78f" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 39685 } } peers { permanent_uuid: "4a0c1377773e425bb07114baf26a4767" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 35839 } } peers { permanent_uuid: "5753549445aa461d80a314d89a2fc823" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 37087 } } }
I20260620 20:12:43.437940 18455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:43.439095 18474 catalog_manager.cc:1347] Generated new cluster ID: d983f1292a8743428ac8a2a4a48f75db
I20260620 20:12:43.439208 18474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:12:43.476318 18491 catalog_manager.cc:1259] Loaded cluster ID: d983f1292a8743428ac8a2a4a48f75db
I20260620 20:12:43.476540 18491 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767: loading cluster ID for follower catalog manager: success
I20260620 20:12:43.480473 18491 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a0c1377773e425bb07114baf26a4767: acquiring CA information for follower catalog manager: success
W20260620 20:12:43.481958 18491 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260620 20:12:43.482460 18474 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:12:43.488585 18474 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:12:43.508174 18474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d1d3ee3da941f5a8f8ec9800ebb78f: Generated new TSK 0
I20260620 20:12:43.514601 18474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:12:44.198639 18452 catalog_manager.cc:1259] Loaded cluster ID: d983f1292a8743428ac8a2a4a48f75db
I20260620 20:12:44.198736 18452 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823: loading cluster ID for follower catalog manager: success
I20260620 20:12:44.201370 18452 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823: acquiring CA information for follower catalog manager: success
I20260620 20:12:44.202558 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 18211
I20260620 20:12:44.203644 18452 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5753549445aa461d80a314d89a2fc823: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:12:44.203859 18273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:44.315975 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 18211
I20260620 20:12:44.336889 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 18285
I20260620 20:12:44.337992 18347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:44.447912 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 18285
I20260620 20:12:44.468483 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 18359
I20260620 20:12:44.469679 18421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:44.577497 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 18359
2026-06-20T20:12:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4309 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:12:44 dist-test-slave-4bwq krb5kdc[18505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:12:44 dist-test-slave-4bwq krb5kdc[18505](info): set up 2 sockets
Jun 20 20:12:44 dist-test-slave-4bwq krb5kdc[18505](info): commencing operation
krb5kdc: starting...
W20260620 20:12:45.699031 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:12:45 dist-test-slave-4bwq krb5kdc[18505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:12:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:12:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:12:45.937860 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:35513
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:46.273651 18518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:46.273973 18518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:46.274149 18518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:46.281621 18518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:12:46.281733 18518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:46.281827 18518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:46.281913 18518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:12:46.282024 18518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:12:46.292111 18518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:35513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:46.293491 18518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:46.294844 18518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:46.307698 18523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:46.307698 18526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:46.307749 18524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:46.308363 18518 server_base.cc:1034] running on GCE node
I20260620 20:12:46.309249 18518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:46.310972 18518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:46.312234 18518 hybrid_clock.cc:648] HybridClock initialized: now 1781986366312182 us; error 38 us; skew 500 ppm
I20260620 20:12:46.312626 18518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:46 dist-test-slave-4bwq krb5kdc[18505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986366, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:46.316061 18518 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:12:46.317589 18518 webserver.cc:460] Webserver started at http://127.4.169.190:45743/ using document root <none> and password file <none>
I20260620 20:12:46.318285 18518 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:46.318441 18518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:46.318737 18518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:12:46.321831 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "8212ec528c0c47d190ef351fe04998e1"
format_stamp: "Formatted at 2026-06-20 20:12:46 on dist-test-slave-4bwq"
server_key: "921c3bcbfc3ee5d6ed6ff3db5cabe1df"
server_key_iv: "58d87fd029e82c4cbafa59736051d263"
server_key_version: "encryptionkey@0"
I20260620 20:12:46.322677 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "8212ec528c0c47d190ef351fe04998e1"
format_stamp: "Formatted at 2026-06-20 20:12:46 on dist-test-slave-4bwq"
server_key: "921c3bcbfc3ee5d6ed6ff3db5cabe1df"
server_key_iv: "58d87fd029e82c4cbafa59736051d263"
server_key_version: "encryptionkey@0"
I20260620 20:12:46.328244 18518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:12:46.331931 18533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:46.333420 18518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:12:46.333578 18518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "8212ec528c0c47d190ef351fe04998e1"
format_stamp: "Formatted at 2026-06-20 20:12:46 on dist-test-slave-4bwq"
server_key: "921c3bcbfc3ee5d6ed6ff3db5cabe1df"
server_key_iv: "58d87fd029e82c4cbafa59736051d263"
server_key_version: "encryptionkey@0"
I20260620 20:12:46.333762 18518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:46.348500 18518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:46.352944 18518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:46.353190 18518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:46.372242 18518 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:35513
I20260620 20:12:46.372267 18584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:35513 every 8 connection(s)
I20260620 20:12:46.373685 18518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:12:46.377564 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:46.380986 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 18518
I20260620 20:12:46.381224 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:12:46.381882 4774 external_mini_cluster.cc:1428] Setting key b83611e1d614cffcc745d9f17681cbf5
I20260620 20:12:46.391160 18585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1: Bootstrap starting.
I20260620 20:12:46.394804 18585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:46.395840 18585 log.cc:824] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:12:46 dist-test-slave-4bwq krb5kdc[18505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:46.398975 18585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1: No bootstrap required, opened a new log
I20260620 20:12:46.407572 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:46.383876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39998 (local address 127.4.169.190:35513)
0620 20:12:46.384700 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:46.384716 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:46.384765 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:12:46.386842 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:46.386874 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:46.387455 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:46.387929 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:46.389288 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:46.391284 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:46.392643 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:46.393709 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:46.393897 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:46.398210 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:46.398220 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:46.398239 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:46.398286 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:46.402439 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:46.403237 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:46.403251 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:46.403261 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:46.403481 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:46.403992 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:46.403998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:46.404002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:46.404770 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:46.405097 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:46.405479 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 20:12:46.406002 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":363,"threads_started":1}
I20260620 20:12:46.409723 18585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8212ec528c0c47d190ef351fe04998e1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35513 } }
I20260620 20:12:46.410149 18585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:46.410283 18585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8212ec528c0c47d190ef351fe04998e1, State: Initialized, Role: FOLLOWER
I20260620 20:12:46.411015 18585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8212ec528c0c47d190ef351fe04998e1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35513 } }
I20260620 20:12:46.411316 18585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:12:46.411527 18585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:12:46.411717 18585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:46.412174 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:12:46.414125 18585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8212ec528c0c47d190ef351fe04998e1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35513 } }
I20260620 20:12:46.414810 18585 leader_election.cc:302] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8212ec528c0c47d190ef351fe04998e1; no voters:
I20260620 20:12:46.415458 18585 leader_election.cc:288] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:12:46.415619 18590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:12:46.416338 18590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8212ec528c0c47d190ef351fe04998e1, State: Running, Role: LEADER
I20260620 20:12:46.416985 18590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8212ec528c0c47d190ef351fe04998e1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35513 } }
I20260620 20:12:46.417675 18585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:12:46.419982 18592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8212ec528c0c47d190ef351fe04998e1. Latest consensus state: current_term: 1 leader_uuid: "8212ec528c0c47d190ef351fe04998e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8212ec528c0c47d190ef351fe04998e1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35513 } } }
I20260620 20:12:46.420410 18592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:46.421020 18591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8212ec528c0c47d190ef351fe04998e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8212ec528c0c47d190ef351fe04998e1" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 35513 } } }
I20260620 20:12:46.421411 18591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:46.423269 18596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:12:46.429724 18596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:12:46.439527 18596 catalog_manager.cc:1347] Generated new cluster ID: c09f942e5a9a46aa9b3a0b09d3de7330
I20260620 20:12:46.439674 18596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:12:46.447636 18596 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:12:46.449028 18596 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:12:46.461414 18596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8212ec528c0c47d190ef351fe04998e1: Generated new TSK 0
I20260620 20:12:46.462333 18596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:12:46.472888 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 18518
I20260620 20:12:46.474179 18580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:46.565070 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 18518
2026-06-20T20:12:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1982 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:12:46 dist-test-slave-4bwq krb5kdc[18613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:12:46 dist-test-slave-4bwq krb5kdc[18613](info): set up 2 sockets
Jun 20 20:12:46 dist-test-slave-4bwq krb5kdc[18613](info): commencing operation
krb5kdc: starting...
W20260620 20:12:47.671279 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:12:47 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:12:47.909762 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45777
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:45777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:48.246508 18626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:48.246897 18626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:48.247076 18626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:48.254644 18626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:12:48.254806 18626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:48.254917 18626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:48.255024 18626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:12:48.255111 18626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:12:48.265306 18626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:45777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:48.266832 18626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:48.268165 18626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:48.280788 18631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:48.280858 18632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:48.280752 18634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:48.281576 18626 server_base.cc:1034] running on GCE node
I20260620 20:12:48.282371 18626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:48.283877 18626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:48.285120 18626 hybrid_clock.cc:648] HybridClock initialized: now 1781986368285072 us; error 38 us; skew 500 ppm
I20260620 20:12:48.285494 18626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:48 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986368, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:48.293627 18626 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:12:48.295408 18626 webserver.cc:460] Webserver started at http://127.4.169.190:45187/ using document root <none> and password file <none>
I20260620 20:12:48.296157 18626 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:48.296319 18626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:48.296643 18626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:12:48.299916 18626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "3c79daeeed124df5821f34eb1c131e50"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "44dfb58d313334df366e9cff04633d72"
server_key_iv: "24d299450aa9412016e869c429a55fc5"
server_key_version: "encryptionkey@0"
I20260620 20:12:48.300786 18626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "3c79daeeed124df5821f34eb1c131e50"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "44dfb58d313334df366e9cff04633d72"
server_key_iv: "24d299450aa9412016e869c429a55fc5"
server_key_version: "encryptionkey@0"
I20260620 20:12:48.306296 18626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:12:48.310184 18641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:48.311756 18626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:12:48.311976 18626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "3c79daeeed124df5821f34eb1c131e50"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "44dfb58d313334df366e9cff04633d72"
server_key_iv: "24d299450aa9412016e869c429a55fc5"
server_key_version: "encryptionkey@0"
I20260620 20:12:48.312228 18626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:48.340014 18626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:48.344893 18626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:48.345136 18626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:48.364115 18626 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:45777
I20260620 20:12:48.364145 18692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:45777 every 8 connection(s)
I20260620 20:12:48.365569 18626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:12:48.369663 18693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:48.373939 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 18626
I20260620 20:12:48.374169 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:12:48.375090 4774 external_mini_cluster.cc:1428] Setting key 6ef59fa71b191ef51c44b6d52e491758
I20260620 20:12:48.381899 18693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Bootstrap starting.
I20260620 20:12:48.385637 18693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:48.386888 18693 log.cc:824] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:48.390578 18693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: No bootstrap required, opened a new log
Jun 20 20:12:48 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:48.401582 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:48.377238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.4.169.190:45777)
0620 20:12:48.378062 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:48.378075 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:48.378116 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:12:48.380308 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:48.380332 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:48.380742 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:48.381075 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:48.383027 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:48.384395 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:48.386262 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:48.387215 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:48.387425 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:48.392442 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:48.392454 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:48.392478 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:48.392531 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:48.396693 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:48.397411 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:48.397420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:48.397430 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:48.397656 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:48.398016 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:48.398027 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:48.398031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:48.398820 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:48.399186 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:48.399519 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:12:48.400049 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":382,"threads_started":1}
I20260620 20:12:48.401775 18693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:48.402242 18693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:48.402369 18693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c79daeeed124df5821f34eb1c131e50, State: Initialized, Role: FOLLOWER
I20260620 20:12:48.402987 18693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:48.403275 18693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:12:48.403447 18693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:12:48.403635 18693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:48.406272 18693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:48.406905 18693 leader_election.cc:302] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c79daeeed124df5821f34eb1c131e50; no voters:
I20260620 20:12:48.407701 18693 leader_election.cc:288] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:12:48.407810 18698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:12:48.408563 18698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 1 LEADER]: Becoming Leader. State: Replica: 3c79daeeed124df5821f34eb1c131e50, State: Running, Role: LEADER
I20260620 20:12:48.409162 18698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:48.410020 18693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:12:48.412026 18699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c79daeeed124df5821f34eb1c131e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } } }
I20260620 20:12:48.412411 18699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:48.413183 18700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c79daeeed124df5821f34eb1c131e50. Latest consensus state: current_term: 1 leader_uuid: "3c79daeeed124df5821f34eb1c131e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } } }
I20260620 20:12:48.413558 18700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:48.415297 18704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:12:48.422029 18704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:12:48.431648 18704 catalog_manager.cc:1347] Generated new cluster ID: 03e2feaa6b9b455d8040ab95c71fe8f3
I20260620 20:12:48.431834 18704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:12:48.447460 18704 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:12:48.449365 18704 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:12:48.457187 18704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Generated new TSK 0
I20260620 20:12:48.458073 18704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:12:48.526522 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:45777
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:48.870980 18719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:48.871272 18719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:48.871454 18719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:48.879163 18719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:48.879278 18719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:48.879474 18719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:12:48.889766 18719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:48.891266 18719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:48.892692 18719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:48.907997 18725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:48.907974 18727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:48.907950 18724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:48.908474 18719 server_base.cc:1034] running on GCE node
I20260620 20:12:48.909403 18719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:48.910600 18719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:48.911842 18719 hybrid_clock.cc:648] HybridClock initialized: now 1781986368911793 us; error 49 us; skew 500 ppm
I20260620 20:12:48.912185 18719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:48 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986368, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:48.915943 18719 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:12:48.917685 18719 webserver.cc:460] Webserver started at http://127.4.169.129:39187/ using document root <none> and password file <none>
I20260620 20:12:48.918710 18719 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:48.918892 18719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:48.919198 18719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:12:48.922772 18719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "c5f9f3abac5f9da993ffa20959dfef03"
server_key_iv: "8d8ee657d97168becaa7d0b976801915"
server_key_version: "encryptionkey@0"
I20260620 20:12:48.923640 18719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "c5f9f3abac5f9da993ffa20959dfef03"
server_key_iv: "8d8ee657d97168becaa7d0b976801915"
server_key_version: "encryptionkey@0"
I20260620 20:12:48.929490 18719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:12:48.933336 18734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:48.934995 18719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:12:48.935155 18719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "c5f9f3abac5f9da993ffa20959dfef03"
server_key_iv: "8d8ee657d97168becaa7d0b976801915"
server_key_version: "encryptionkey@0"
I20260620 20:12:48.935343 18719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:48.954186 18719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:48.959056 18719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:48.959438 18719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:48.960506 18719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:48.961951 18719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:48.962031 18719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:48.962198 18719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:48.962266 18719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:48.997582 18719 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:42943
I20260620 20:12:48.997687 18846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:42943 every 8 connection(s)
I20260620 20:12:48.999003 18719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:12:49.000789 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 18719
I20260620 20:12:49.001011 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:12:49.001745 4774 external_mini_cluster.cc:1428] Setting key efd3d9818675b783b9d5882373f5c529
Jun 20 20:12:49 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986368, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:49.025400 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:49.001930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:36705 (local address 127.4.169.190:45777)
0620 20:12:49.002481 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:49.002488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:49.003678 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 20:12:49.006553 (+ 2875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:49.006558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:49.006629 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:49.006733 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:49.009144 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:49.009910 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:49.012571 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:49.013122 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:49.013279 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:49.017475 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:49.017485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:49.017513 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:49.017567 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:49.021445 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:49.022354 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:49.022363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:49.022369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:49.022582 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:49.023119 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:49.023127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:49.023134 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:49.023561 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:49.023736 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:49.024634 (+ 898us) server_negotiation.cc:299] Negotiation successful
0620 20:12:49.024933 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":251,"threads_started":1}
I20260620 20:12:49.026351 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:49.002626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:36705)
0620 20:12:49.003498 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:49.003553 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:49.006163 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:49.007101 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:49.007116 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:49.008000 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:49.008916 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:49.008949 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:49.010096 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:49.010104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:49.012372 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:49.012394 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:49.013367 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:49.013372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:49.013583 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:49.014350 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:49.014428 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:49.017187 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:49.021654 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:49.021662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:49.021678 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:49.022191 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:49.022729 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:49.022733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:49.022735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:49.022980 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:49.023748 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:49.023756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:49.024183 (+ 427us) client_negotiation.cc:769] Sending connection context
0620 20:12:49.024461 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 20:12:49.025034 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":340,"threads_started":1}
I20260620 20:12:49.028221 18847 heartbeater.cc:347] Connected to a master server at 127.4.169.190:45777
I20260620 20:12:49.028632 18847 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:12:49.029723 18847 heartbeater.cc:511] Master 127.4.169.190:45777 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
I20260620 20:12:49.032734 18658 ts_manager.cc:194] Registered new tserver with Master: e529e4c6b85e4cd19db1b5adf6a5f9a2 (127.4.169.129:42943)
I20260620 20:12:49.035070 18658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:36705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:12:49.061597 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:45777
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:49.401947 18853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:49.402246 18853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:49.402464 18853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:49.410025 18853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:49.410152 18853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:49.410356 18853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:12:49.420643 18853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:49.421988 18853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:49.423441 18853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:49.436965 18858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:49.436908 18861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:49.436884 18859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:49.437247 18853 server_base.cc:1034] running on GCE node
I20260620 20:12:49.438071 18853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:49.439169 18853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:49.440442 18853 hybrid_clock.cc:648] HybridClock initialized: now 1781986369440415 us; error 55 us; skew 500 ppm
I20260620 20:12:49.440889 18853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:49 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986369, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:49.444370 18853 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:12:49.445937 18853 webserver.cc:460] Webserver started at http://127.4.169.130:39949/ using document root <none> and password file <none>
I20260620 20:12:49.446734 18853 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:49.446897 18853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:49.447212 18853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:12:49.450186 18853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "4061fba506d04bd2ad541bf3563d88cd"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "ce690fcaf504f45c28d264f7e324daac"
server_key_iv: "e7529f69f712a34abc7084706881c9b0"
server_key_version: "encryptionkey@0"
I20260620 20:12:49.451069 18853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "4061fba506d04bd2ad541bf3563d88cd"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "ce690fcaf504f45c28d264f7e324daac"
server_key_iv: "e7529f69f712a34abc7084706881c9b0"
server_key_version: "encryptionkey@0"
I20260620 20:12:49.456493 18853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:12:49.460148 18868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:49.461783 18853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:12:49.462020 18853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "4061fba506d04bd2ad541bf3563d88cd"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "ce690fcaf504f45c28d264f7e324daac"
server_key_iv: "e7529f69f712a34abc7084706881c9b0"
server_key_version: "encryptionkey@0"
I20260620 20:12:49.462276 18853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:49.494915 18853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:49.499512 18853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:49.499830 18853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:49.500835 18853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:49.502362 18853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:49.502522 18853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 20:12:49.502684 18853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:49.502759 18853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:49.537078 18853 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:42007
I20260620 20:12:49.537122 18980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:42007 every 8 connection(s)
I20260620 20:12:49.538560 18853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:12:49.545737 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 18853
I20260620 20:12:49.545934 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:12:49.546442 4774 external_mini_cluster.cc:1428] Setting key e44325e0df2ede7602f84eddc90ef086
Jun 20 20:12:49 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986369, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:49.560153 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:49.541378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:47927 (local address 127.4.169.190:45777)
0620 20:12:49.541797 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:49.541802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:49.542933 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 20:12:49.544878 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:49.544885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:49.544985 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:49.545110 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:49.546940 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:49.547866 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:49.549465 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:49.549871 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:49.550011 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:49.553684 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:49.553689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:49.553692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:49.553727 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:49.556460 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:49.557486 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:49.557491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:49.557493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:49.557645 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:49.558179 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:49.558183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:49.558185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:49.558551 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:49.558672 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:49.559658 (+ 986us) server_negotiation.cc:299] Negotiation successful
0620 20:12:49.559860 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":310,"thread_start_us":200,"threads_started":1}
I20260620 20:12:49.561705 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:49.542002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:47927)
0620 20:12:49.542761 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:49.542799 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:49.544563 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:49.545366 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:49.545375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:49.546004 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:49.546755 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:49.546777 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:49.548031 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:49.548035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:49.549303 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:49.549316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:49.550111 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:49.550118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:49.550354 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:49.551167 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:49.551190 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:49.553436 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:49.556615 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:49.556627 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:49.556649 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:49.557310 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:49.557767 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:49.557775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:49.557781 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:49.558049 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:49.558679 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:49.558690 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:49.559218 (+ 528us) client_negotiation.cc:769] Sending connection context
0620 20:12:49.559544 (+ 326us) client_negotiation.cc:240] Negotiation successful
0620 20:12:49.560178 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":344,"threads_started":1}
I20260620 20:12:49.563412 18981 heartbeater.cc:347] Connected to a master server at 127.4.169.190:45777
I20260620 20:12:49.563751 18981 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:49.564564 18981 heartbeater.cc:511] Master 127.4.169.190:45777 requested a full tablet report, sending...
I20260620 20:12:49.566184 18658 ts_manager.cc:194] Registered new tserver with Master: 4061fba506d04bd2ad541bf3563d88cd (127.4.169.130:42007)
I20260620 20:12:49.567245 18658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:47927
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:12:49.602066 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:45777
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:49.938777 18987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:49.939169 18987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:49.939332 18987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:49.947121 18987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:49.947247 18987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:49.947467 18987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:12:49.957626 18987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:49.959079 18987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:49.960496 18987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:49.974890 18995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:49.974939 18992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:49.974869 18993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:49.975826 18987 server_base.cc:1034] running on GCE node
I20260620 20:12:49.976507 18987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:49.977515 18987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:49.978757 18987 hybrid_clock.cc:648] HybridClock initialized: now 1781986369978696 us; error 45 us; skew 500 ppm
I20260620 20:12:49.979146 18987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:49 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986369, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:49.982487 18987 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:12:49.984081 18987 webserver.cc:460] Webserver started at http://127.4.169.131:33215/ using document root <none> and password file <none>
I20260620 20:12:49.984773 18987 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:49.984876 18987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:49.985126 18987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:12:49.988204 18987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "fd3bab861b0d47f8a849d71c0d61041a"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "95fc1e229785c64f1233a3921fb5277d"
server_key_iv: "a10f20d3119b2b5bdd819d3346c00d39"
server_key_version: "encryptionkey@0"
I20260620 20:12:49.988976 18987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "fd3bab861b0d47f8a849d71c0d61041a"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "95fc1e229785c64f1233a3921fb5277d"
server_key_iv: "a10f20d3119b2b5bdd819d3346c00d39"
server_key_version: "encryptionkey@0"
I20260620 20:12:49.994269 18987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:12:49.998181 19002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:49.999723 18987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:12:49.999892 18987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "fd3bab861b0d47f8a849d71c0d61041a"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "95fc1e229785c64f1233a3921fb5277d"
server_key_iv: "a10f20d3119b2b5bdd819d3346c00d39"
server_key_version: "encryptionkey@0"
I20260620 20:12:50.000105 18987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:50.020372 18987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:50.024775 18987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:50.025094 18987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:50.026117 18987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:50.027616 18987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:50.027693 18987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:50.027845 18987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:50.027918 18987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:50.038697 18847 heartbeater.cc:503] Master 127.4.169.190:45777 was elected leader, sending a full tablet report...
I20260620 20:12:50.076138 18987 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:38967
I20260620 20:12:50.076188 19114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:38967 every 8 connection(s)
I20260620 20:12:50.077646 18987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:12:50.086006 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 18987
I20260620 20:12:50.086251 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:12:50.086968 4774 external_mini_cluster.cc:1428] Setting key bfd63408bdafec65381989b8359f0d57
Jun 20 20:12:50 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986369, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:50.102383 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.080999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:53769 (local address 127.4.169.190:45777)
0620 20:12:50.081547 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.081555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.082822 (+ 1267us) server_negotiation.cc:407] Connection header received
0620 20:12:50.084801 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.084807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.084917 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.085059 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:50.087507 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.088686 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.090750 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.091177 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.091323 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.095738 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:50.095744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:50.095748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:50.095790 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:50.098759 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.099667 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.099672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.099675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.099811 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.100281 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.100284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.100286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.100674 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:50.100801 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.101738 (+ 937us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.102038 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":239,"threads_started":1}
I20260620 20:12:50.103792 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.081664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:53769)
0620 20:12:50.082589 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:50.082648 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:50.084455 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:50.085442 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:50.085482 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:50.086270 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:50.087282 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.087306 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.088877 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.088881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.090582 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.090597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.091430 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.091438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.091685 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.092371 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:50.092402 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:50.095386 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:50.098947 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:50.098956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:50.098971 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:50.099518 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:50.099957 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:50.099961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:50.099964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:50.100170 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:50.100827 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:50.100836 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:50.101312 (+ 476us) client_negotiation.cc:769] Sending connection context
0620 20:12:50.101604 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 20:12:50.102312 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":362,"threads_started":1}
I20260620 20:12:50.105368 19115 heartbeater.cc:347] Connected to a master server at 127.4.169.190:45777
I20260620 20:12:50.105669 19115 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:50.106621 19115 heartbeater.cc:511] Master 127.4.169.190:45777 requested a full tablet report, sending...
I20260620 20:12:50.108300 18658 ts_manager.cc:194] Registered new tserver with Master: fd3bab861b0d47f8a849d71c0d61041a (127.4.169.131:38967)
I20260620 20:12:50.109225 18658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:53769
I20260620 20:12:50.115806 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:12:50.134128 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.120863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.4.169.190:45777)
0620 20:12:50.121096 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.121104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.121468 (+ 364us) server_negotiation.cc:407] Connection header received
0620 20:12:50.121676 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.121682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.121760 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.121864 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:50.123325 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.124071 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.125412 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.125834 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.125976 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.128599 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:50.128605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:50.128608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:50.128645 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:50.131428 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.132121 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.132128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.132132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.132267 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.132640 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.132648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.132653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.132984 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:50.133247 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.133563 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.133802 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:12:50.143221 18658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:12:50.150517 18658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:12:50.175585 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.163393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:42007 (local address 127.0.0.1:52508)
0620 20:12:50.165119 (+ 1726us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:50.165172 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:50.165345 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:50.167127 (+ 1782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:50.167134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:50.167173 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:50.167652 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.167669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.169699 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.169704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.172220 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.172236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.174534 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.174541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.174733 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.174792 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:12:50.174945 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 20:12:50.175154 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1449,"thread_start_us":3104,"threads_started":1}
I20260620 20:12:50.176286 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.164063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:38967 (local address 127.0.0.1:56300)
0620 20:12:50.166791 (+ 2728us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:50.166816 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:50.166926 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:50.168753 (+ 1827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:50.168758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:50.168785 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:50.169186 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.169198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.171157 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.171162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.173337 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.173354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.175461 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.175467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.175676 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.175716 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 20:12:50.175811 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 20:12:50.175934 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2581,"thread_start_us":913,"threads_started":1}
I20260620 20:12:50.177043 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.163528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.4.169.130:42007)
0620 20:12:50.166006 (+ 2478us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.166019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.166056 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:12:50.166163 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.166171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.166468 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.166907 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:50.167999 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.169545 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.172448 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.174019 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.174167 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.174381 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.176511 (+ 2130us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.176706 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2205,"thread_start_us":2093,"threads_started":1}
I20260620 20:12:50.177605 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.164693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.4.169.131:38967)
0620 20:12:50.166535 (+ 1842us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.166543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.167048 (+ 505us) server_negotiation.cc:407] Connection header received
0620 20:12:50.167190 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.167200 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.167516 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.167702 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:50.169388 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.170991 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.173534 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.174981 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.175090 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.175302 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.177048 (+ 1746us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.177237 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1597,"thread_start_us":1415,"threads_started":1}
I20260620 20:12:50.181860 18916 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca03c95ee434c5281b78f5951009d6a (DEFAULT_TABLE table=test-table [id=2fe1196eb16f4c02bc19f6e1bb7a00f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:12:50.181861 19050 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca03c95ee434c5281b78f5951009d6a (DEFAULT_TABLE table=test-table [id=2fe1196eb16f4c02bc19f6e1bb7a00f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:12:50.183249 19050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca03c95ee434c5281b78f5951009d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:50.183290 18916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca03c95ee434c5281b78f5951009d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:50.185395 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.168099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:41972)
0620 20:12:50.176175 (+ 8076us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:50.176201 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:50.176312 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:50.178711 (+ 2399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:50.178716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:50.178748 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:50.178918 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.178927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.180821 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.180827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.182679 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.182696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.184586 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.184592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.184766 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.184799 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:12:50.184912 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:12:50.185092 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7972,"thread_start_us":10017,"threads_started":1}
I20260620 20:12:50.185490 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.167262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.4.169.129:42943)
0620 20:12:50.178026 (+ 10764us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.178036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.178067 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:12:50.178176 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.178184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.178494 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.178709 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:50.179083 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.180628 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.182859 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.184382 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.184530 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.184763 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.184971 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.185150 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10569,"thread_start_us":10377,"threads_started":1}
I20260620 20:12:50.189954 18782 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca03c95ee434c5281b78f5951009d6a (DEFAULT_TABLE table=test-table [id=2fe1196eb16f4c02bc19f6e1bb7a00f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:12:50.191581 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca03c95ee434c5281b78f5951009d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:50.196017 19134 tablet_bootstrap.cc:492] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: Bootstrap starting.
I20260620 20:12:50.198863 19135 tablet_bootstrap.cc:492] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: Bootstrap starting.
I20260620 20:12:50.199883 19134 tablet_bootstrap.cc:654] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:50.201308 19134 log.cc:824] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:50.202621 19135 tablet_bootstrap.cc:654] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:50.203914 19135 log.cc:824] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:50.204715 19134 tablet_bootstrap.cc:492] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: No bootstrap required, opened a new log
I20260620 20:12:50.205065 19134 ts_tablet_manager.cc:1397] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 20:12:50.206840 19138 tablet_bootstrap.cc:492] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: Bootstrap starting.
I20260620 20:12:50.207814 19135 tablet_bootstrap.cc:492] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: No bootstrap required, opened a new log
I20260620 20:12:50.208117 19135 ts_tablet_manager.cc:1397] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 20:12:50.210851 19138 tablet_bootstrap.cc:654] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:50.212214 19138 log.cc:824] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:50.213518 19134 raft_consensus.cc:348] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } }
I20260620 20:12:50.214082 19134 raft_consensus.cc:374] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:50.214222 19134 raft_consensus.cc:729] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4061fba506d04bd2ad541bf3563d88cd, State: Initialized, Role: FOLLOWER
I20260620 20:12:50.214979 19134 consensus_queue.cc:260] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } }
I20260620 20:12:50.216320 18981 heartbeater.cc:503] Master 127.4.169.190:45777 was elected leader, sending a full tablet report...
I20260620 20:12:50.216377 19138 tablet_bootstrap.cc:492] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: No bootstrap required, opened a new log
I20260620 20:12:50.216747 19138 ts_tablet_manager.cc:1397] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 20:12:50.217231 19134 ts_tablet_manager.cc:1428] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 20:12:50.218930 19135 raft_consensus.cc:348] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } }
I20260620 20:12:50.219414 19135 raft_consensus.cc:374] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:50.219521 19135 raft_consensus.cc:729] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3bab861b0d47f8a849d71c0d61041a, State: Initialized, Role: FOLLOWER
I20260620 20:12:50.220202 19135 consensus_queue.cc:260] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } }
I20260620 20:12:50.224619 19115 heartbeater.cc:503] Master 127.4.169.190:45777 was elected leader, sending a full tablet report...
I20260620 20:12:50.225205 19135 ts_tablet_manager.cc:1428] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260620 20:12:50.226006 19138 raft_consensus.cc:348] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } }
I20260620 20:12:50.226529 19138 raft_consensus.cc:374] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:50.226630 19138 raft_consensus.cc:729] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e529e4c6b85e4cd19db1b5adf6a5f9a2, State: Initialized, Role: FOLLOWER
I20260620 20:12:50.227236 19138 consensus_queue.cc:260] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } }
I20260620 20:12:50.229194 19138 ts_tablet_manager.cc:1428] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260620 20:12:50.253003 18848 tablet.cc:2369] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:12:50.290825 18982 tablet.cc:2369] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:12:50.303483 19140 raft_consensus.cc:482] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:12:50.303910 19140 raft_consensus.cc:504] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } }
I20260620 20:12:50.305696 19140 leader_election.cc:288] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e529e4c6b85e4cd19db1b5adf6a5f9a2 (127.4.169.129:42943), fd3bab861b0d47f8a849d71c0d61041a (127.4.169.131:38967)
I20260620 20:12:50.315368 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.306286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:38047 (local address 127.4.169.129:42943)
0620 20:12:50.306604 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.306611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.307092 (+ 481us) server_negotiation.cc:407] Connection header received
0620 20:12:50.307294 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.307298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.307411 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.307533 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:50.308314 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.309466 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.312593 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.313855 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.314048 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.314128 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.314951 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.315098 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260620 20:12:50.315361 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.306159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:35443 (local address 127.4.169.131:38967)
0620 20:12:50.306400 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.306431 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.307348 (+ 917us) server_negotiation.cc:407] Connection header received
0620 20:12:50.307506 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.307515 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.307592 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.307683 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:50.308502 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.309586 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.312584 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.313854 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.314052 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.314128 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.314948 (+ 820us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.315098 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 20:12:50.315575 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.306049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.4.169.130:38047)
0620 20:12:50.306946 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:50.306972 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:50.307156 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:50.307556 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:50.307560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:50.307580 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:50.308147 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.308163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.309622 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.309627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.312393 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.312412 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.314217 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.314223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.314589 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.314748 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 20:12:50.314863 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:12:50.315023 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":258,"spinlock_wait_cycles":22784,"thread_start_us":859,"threads_started":1}
I20260620 20:12:50.315714 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.306069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:38967 (local address 127.4.169.130:35443)
0620 20:12:50.307244 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:50.307267 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:50.307379 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:50.307713 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:50.307719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:50.307806 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:50.308339 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.308349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.309742 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.309750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.312393 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:50.312412 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.314197 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.314208 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:50.314588 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.314744 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 20:12:50.314934 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 20:12:50.315037 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":325,"thread_start_us":457,"threads_started":1}
I20260620 20:12:50.316910 19070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca03c95ee434c5281b78f5951009d6a" candidate_uuid: "4061fba506d04bd2ad541bf3563d88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3bab861b0d47f8a849d71c0d61041a" is_pre_election: true
I20260620 20:12:50.316964 18802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca03c95ee434c5281b78f5951009d6a" candidate_uuid: "4061fba506d04bd2ad541bf3563d88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" is_pre_election: true
I20260620 20:12:50.317468 19070 raft_consensus.cc:2454] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4061fba506d04bd2ad541bf3563d88cd in term 0.
I20260620 20:12:50.317582 18802 raft_consensus.cc:2454] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4061fba506d04bd2ad541bf3563d88cd in term 0.
I20260620 20:12:50.318369 18869 leader_election.cc:302] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4061fba506d04bd2ad541bf3563d88cd, e529e4c6b85e4cd19db1b5adf6a5f9a2, fd3bab861b0d47f8a849d71c0d61041a; no voters:
I20260620 20:12:50.318900 19140 raft_consensus.cc:2781] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:12:50.319054 19140 raft_consensus.cc:482] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:12:50.319187 19140 raft_consensus.cc:3037] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:50.321434 19140 raft_consensus.cc:504] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } }
I20260620 20:12:50.322054 19140 leader_election.cc:288] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [CANDIDATE]: Term 1 election: Requested vote from peers e529e4c6b85e4cd19db1b5adf6a5f9a2 (127.4.169.129:42943), fd3bab861b0d47f8a849d71c0d61041a (127.4.169.131:38967)
I20260620 20:12:50.322609 18802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca03c95ee434c5281b78f5951009d6a" candidate_uuid: "4061fba506d04bd2ad541bf3563d88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2"
I20260620 20:12:50.322628 19070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca03c95ee434c5281b78f5951009d6a" candidate_uuid: "4061fba506d04bd2ad541bf3563d88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3bab861b0d47f8a849d71c0d61041a"
I20260620 20:12:50.322850 18802 raft_consensus.cc:3037] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:50.322849 19070 raft_consensus.cc:3037] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:50.326174 18802 raft_consensus.cc:2454] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4061fba506d04bd2ad541bf3563d88cd in term 1.
I20260620 20:12:50.326174 19070 raft_consensus.cc:2454] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4061fba506d04bd2ad541bf3563d88cd in term 1.
I20260620 20:12:50.326749 18871 leader_election.cc:302] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4061fba506d04bd2ad541bf3563d88cd, e529e4c6b85e4cd19db1b5adf6a5f9a2; no voters:
I20260620 20:12:50.327172 19140 raft_consensus.cc:2781] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:12:50.327706 19140 raft_consensus.cc:686] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 1 LEADER]: Becoming Leader. State: Replica: 4061fba506d04bd2ad541bf3563d88cd, State: Running, Role: LEADER
I20260620 20:12:50.328208 19140 consensus_queue.cc:237] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } }
W20260620 20:12:50.329602 19116 tablet.cc:2369] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:12:50.333673 18658 catalog_manager.cc:5495] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4061fba506d04bd2ad541bf3563d88cd (127.4.169.130). New cstate: current_term: 1 leader_uuid: "4061fba506d04bd2ad541bf3563d88cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260620 20:12:50.370582 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.364023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.4.169.130:42007)
0620 20:12:50.364270 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.364279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.364327 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:12:50.364431 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.364437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.364542 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.364699 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:12:50.365116 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.366199 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.367885 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.368557 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.368684 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.369352 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:12:50.369894 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:12:50.370063 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.370168 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.370259 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:12:50.379904 19070 raft_consensus.cc:1261] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [term 1 FOLLOWER]: Refusing update from remote peer 4061fba506d04bd2ad541bf3563d88cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:12:50.379901 18802 raft_consensus.cc:1261] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 1 FOLLOWER]: Refusing update from remote peer 4061fba506d04bd2ad541bf3563d88cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:12:50.380976 19145 consensus_queue.cc:1035] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:12:50.381359 19140 consensus_queue.cc:1035] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:12:50.395113 19148 mvcc.cc:205] Tried to move back new op lower bound from 7299016173065265152 to 7299016172868620288. Current Snapshot: MvccSnapshot[applied={T|T < 7299016173065265152}]
I20260620 20:12:50.410173 18657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:12:50.410550 18657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:12:50.415503 18657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Sending DeleteTablet for 3 replicas of tablet 0ca03c95ee434c5281b78f5951009d6a
I20260620 20:12:50.417268 18782 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca03c95ee434c5281b78f5951009d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:12:50 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:41972
I20260620 20:12:50.417330 19050 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca03c95ee434c5281b78f5951009d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:12:50 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:56300
I20260620 20:12:50.417493 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 18719
I20260620 20:12:50.418500 19158 tablet_replica.cc:330] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: stopping tablet replica
I20260620 20:12:50.419160 19158 raft_consensus.cc:2229] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:12:50.419843 19158 raft_consensus.cc:2258] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:12:50.419804 18916 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca03c95ee434c5281b78f5951009d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:12:50 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:52508
I20260620 20:12:50.422761 19158 ts_tablet_manager.cc:1905] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:12:50.430568 19158 ts_tablet_manager.cc:1918] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:12:50.430882 19158 log.cc:1196] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/0ca03c95ee434c5281b78f5951009d6a
I20260620 20:12:50.431586 19158 ts_tablet_manager.cc:1939] T 0ca03c95ee434c5281b78f5951009d6a P e529e4c6b85e4cd19db1b5adf6a5f9a2: Deleting consensus metadata
I20260620 20:12:50.433497 19161 tablet_replica.cc:330] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: stopping tablet replica
I20260620 20:12:50.434052 19161 raft_consensus.cc:2229] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:12:50.434162 18644 catalog_manager.cc:4841] TS e529e4c6b85e4cd19db1b5adf6a5f9a2 (127.4.169.129:42943): tablet 0ca03c95ee434c5281b78f5951009d6a (table test-table [id=2fe1196eb16f4c02bc19f6e1bb7a00f8]) successfully deleted
I20260620 20:12:50.434769 19161 raft_consensus.cc:2258] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:12:50.435459 19159 tablet_replica.cc:330] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: stopping tablet replica
I20260620 20:12:50.435913 19159 raft_consensus.cc:2229] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:12:50.436374 19159 raft_consensus.cc:2258] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:12:50.437034 19161 ts_tablet_manager.cc:1905] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:12:50.439252 19159 ts_tablet_manager.cc:1905] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:12:50.444970 19161 ts_tablet_manager.cc:1918] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:12:50.445194 19161 log.cc:1196] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/0ca03c95ee434c5281b78f5951009d6a
I20260620 20:12:50.445757 19161 ts_tablet_manager.cc:1939] T 0ca03c95ee434c5281b78f5951009d6a P 4061fba506d04bd2ad541bf3563d88cd: Deleting consensus metadata
Jun 20 20:12:50 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:12:50.446676 19159 ts_tablet_manager.cc:1918] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:12:50.446916 19159 log.cc:1196] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/0ca03c95ee434c5281b78f5951009d6a
I20260620 20:12:50.447469 19159 ts_tablet_manager.cc:1939] T 0ca03c95ee434c5281b78f5951009d6a P fd3bab861b0d47f8a849d71c0d61041a: Deleting consensus metadata
I20260620 20:12:50.447983 18644 catalog_manager.cc:4841] TS 4061fba506d04bd2ad541bf3563d88cd (127.4.169.130:42007): tablet 0ca03c95ee434c5281b78f5951009d6a (table test-table [id=2fe1196eb16f4c02bc19f6e1bb7a00f8]) successfully deleted
I20260620 20:12:50.449393 18642 catalog_manager.cc:4841] TS fd3bab861b0d47f8a849d71c0d61041a (127.4.169.131:38967): tablet 0ca03c95ee434c5281b78f5951009d6a (table test-table [id=2fe1196eb16f4c02bc19f6e1bb7a00f8]) successfully deleted
I20260620 20:12:50.454397 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.419034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.4.169.129:42943)
0620 20:12:50.419238 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.419249 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.428308 (+ 9059us) server_negotiation.cc:407] Connection header received
0620 20:12:50.428570 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.428576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.428670 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.428808 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:50.436962 (+ 8154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.437819 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.440571 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.441198 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.441318 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.447618 (+ 6300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:50.447640 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:50.447661 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:50.447716 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:50.451219 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.452198 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.452205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.452207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.452337 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.452862 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.452866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.452869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.453271 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:50.453505 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.453874 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.454066 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:12:50.455088 18842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:50.552784 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 18719
I20260620 20:12:50.570740 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 18853
Jun 20 20:12:50 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:12:50.590041 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.571477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.4.169.130:42007)
0620 20:12:50.571664 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.571669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.571775 (+ 106us) server_negotiation.cc:407] Connection header received
0620 20:12:50.571984 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.571988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.572054 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.572139 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:50.573805 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.574824 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.576828 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.577293 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.577392 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.582676 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:50.582696 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:50.582717 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:50.582789 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:50.586878 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.587755 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.587764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.587768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.587932 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.588338 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.588344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.588347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.588813 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:50.589158 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.589361 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.589640 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:12:50.590991 18976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:50.706851 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 18853
I20260620 20:12:50.727515 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 18987
Jun 20 20:12:50 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:12:50.747110 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:50.728532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.4.169.131:38967)
0620 20:12:50.728723 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:50.728729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:50.728801 (+ 72us) server_negotiation.cc:407] Connection header received
0620 20:12:50.729046 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:50.729052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:50.729130 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:50.729230 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:50.731483 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.732487 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.734375 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:50.735085 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:50.735342 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:50.741164 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:50.741181 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:50.741197 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:50.741240 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:50.744082 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.744883 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.744892 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.744894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.745059 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:50.745429 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:50.745434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:50.745436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:50.745839 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:50.746095 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:50.746585 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 20:12:50.746827 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:12:50.747759 19110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:50.838788 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 18987
I20260620 20:12:50.855790 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 18626
I20260620 20:12:50.857012 18688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:50.968302 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 18626
I20260620 20:12:50.989886 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45777
--webserver_interface=127.4.169.190
--webserver_port=45187
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:45777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:51.333034 19166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:51.333393 19166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:51.333571 19166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:51.340986 19166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:12:51.341142 19166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:51.341246 19166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:51.341351 19166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:12:51.341430 19166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:12:51.351620 19166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:45777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=45187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:51.353026 19166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:51.354375 19166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:51.378473 19173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:51.378469 19172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:51.378444 19175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:51.379297 19166 server_base.cc:1034] running on GCE node
I20260620 20:12:51.380148 19166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:51.381732 19166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:51.382946 19166 hybrid_clock.cc:648] HybridClock initialized: now 1781986371382879 us; error 62 us; skew 500 ppm
I20260620 20:12:51.383383 19166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:51 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:51.387089 19166 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:12:51.388814 19166 webserver.cc:460] Webserver started at http://127.4.169.190:45187/ using document root <none> and password file <none>
I20260620 20:12:51.389597 19166 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:51.389770 19166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:51.395893 19166 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 20:12:51.398952 19182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:51.400426 19166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:12:51.400650 19166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "3c79daeeed124df5821f34eb1c131e50"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "44dfb58d313334df366e9cff04633d72"
server_key_iv: "24d299450aa9412016e869c429a55fc5"
server_key_version: "encryptionkey@0"
I20260620 20:12:51.401767 19166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:51.425062 19166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:51.429694 19166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:51.430101 19166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:51.450049 19166 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:45777
I20260620 20:12:51.450062 19233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:45777 every 8 connection(s)
I20260620 20:12:51.451785 19166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:12:51.453742 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 19166
I20260620 20:12:51.454610 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:42943
--local_ip_for_outbound_sockets=127.4.169.129
--tserver_master_addrs=127.4.169.190:45777
--webserver_port=39187
--webserver_interface=127.4.169.129
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:12:51.459477 19234 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:12:51.463856 19234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Bootstrap starting.
I20260620 20:12:51.490859 19234 log.cc:824] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:51.503675 19234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:12:51.504289 19234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Bootstrap complete.
I20260620 20:12:51.514210 19234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:51.515240 19234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c79daeeed124df5821f34eb1c131e50, State: Initialized, Role: FOLLOWER
I20260620 20:12:51.515987 19234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:51.516359 19234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:12:51.516528 19234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:12:51.516724 19234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:12:51.521139 19234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:51.521740 19234 leader_election.cc:302] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c79daeeed124df5821f34eb1c131e50; no voters:
I20260620 20:12:51.522450 19234 leader_election.cc:288] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:12:51.522629 19238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:12:51.523973 19238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 2 LEADER]: Becoming Leader. State: Replica: 3c79daeeed124df5821f34eb1c131e50, State: Running, Role: LEADER
I20260620 20:12:51.524580 19238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:51.525013 19234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:12:51.527455 19240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c79daeeed124df5821f34eb1c131e50. Latest consensus state: current_term: 2 leader_uuid: "3c79daeeed124df5821f34eb1c131e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } } }
I20260620 20:12:51.527932 19240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:51.528494 19239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3c79daeeed124df5821f34eb1c131e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } } }
I20260620 20:12:51.528862 19239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:51.532517 19244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:12:51.541316 19244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:12:51.542974 19244 catalog_manager.cc:1259] Loaded cluster ID: 03e2feaa6b9b455d8040ab95c71fe8f3
I20260620 20:12:51.543167 19244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:12:51.546594 19244 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 20:12:51.565553 19244 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 20:12:51.582630 19244 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:12:51.584728 19244 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Loaded TSK: 0
I20260620 20:12:51.586774 19244 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:12:51.834028 19236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:51.834481 19236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:51.834661 19236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:51.842576 19236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:51.842739 19236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:51.842919 19236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:12:51.853418 19236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:42943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=39187
--tserver_master_addrs=127.4.169.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:51.854903 19236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:51.856406 19236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:51.872234 19263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:51.872263 19262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:51.872367 19265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:51.873624 19236 server_base.cc:1034] running on GCE node
I20260620 20:12:51.874363 19236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:51.875499 19236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:51.876725 19236 hybrid_clock.cc:648] HybridClock initialized: now 1781986371876675 us; error 45 us; skew 500 ppm
I20260620 20:12:51.877100 19236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:51 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:51.880753 19236 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:12:51.882736 19236 webserver.cc:460] Webserver started at http://127.4.169.129:39187/ using document root <none> and password file <none>
I20260620 20:12:51.883674 19236 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:51.883821 19236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:51.890698 19236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.003s
I20260620 20:12:51.894116 19272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:51.895816 19236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:12:51.896000 19236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "c5f9f3abac5f9da993ffa20959dfef03"
server_key_iv: "8d8ee657d97168becaa7d0b976801915"
server_key_version: "encryptionkey@0"
I20260620 20:12:51.897033 19236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:51.913628 19236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:51.918071 19236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:51.918366 19236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:51.919466 19236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:51.920897 19236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:51.921020 19236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:51.921154 19236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:51.921243 19236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:51.954718 19236 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:42943
I20260620 20:12:51.954804 19384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:42943 every 8 connection(s)
I20260620 20:12:51.956305 19236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:12:51.962960 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 19236
I20260620 20:12:51.963706 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:42007
--local_ip_for_outbound_sockets=127.4.169.130
--tserver_master_addrs=127.4.169.190:45777
--webserver_port=39949
--webserver_interface=127.4.169.130
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:12:51 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:51.983417 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:51.959448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:44775)
0620 20:12:51.960305 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:51.960363 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:51.962130 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:51.963669 (+ 1539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:51.963682 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:51.964517 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:51.965352 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:51.965382 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:51.967159 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:51.967168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:51.969395 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:51.969416 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:51.970537 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:51.970547 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:51.970884 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:51.971673 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:51.971696 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:51.974175 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:51.979016 (+ 4841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:51.979023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:51.979036 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:51.979508 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:51.980034 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:51.980037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:51.980040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:51.980240 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:51.981224 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:51.981230 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:51.981547 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 20:12:51.981817 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 20:12:51.982231 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":353,"threads_started":1}
I20260620 20:12:51.983947 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:51.959090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:44775 (local address 127.4.169.190:45777)
0620 20:12:51.959681 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:51.959693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:51.960486 (+ 793us) server_negotiation.cc:407] Connection header received
0620 20:12:51.962695 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:51.962709 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:51.963107 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:51.963449 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:51.965607 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:51.966988 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:51.969567 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:51.970249 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:51.970472 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:51.974525 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:51.974538 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:51.974561 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:51.974617 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:51.978813 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:51.979660 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:51.979672 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:51.979682 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:51.979906 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:51.980354 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:51.980361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:51.980367 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:51.981066 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:51.981272 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:51.982081 (+ 809us) server_negotiation.cc:299] Negotiation successful
0620 20:12:51.982572 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":267,"threads_started":1}
I20260620 20:12:51.986979 19385 heartbeater.cc:347] Connected to a master server at 127.4.169.190:45777
I20260620 20:12:51.987351 19385 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:51.988355 19385 heartbeater.cc:511] Master 127.4.169.190:45777 requested a full tablet report, sending...
I20260620 20:12:51.990931 19199 ts_manager.cc:194] Registered new tserver with Master: e529e4c6b85e4cd19db1b5adf6a5f9a2 (127.4.169.129:42943)
I20260620 20:12:51.992437 19199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:44775
W20260620 20:12:52.304603 19389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:52.304981 19389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:52.305083 19389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:52.312615 19389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:52.312734 19389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:52.312852 19389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:12:52.323057 19389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:42007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=39949
--tserver_master_addrs=127.4.169.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:52.324342 19389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:52.325678 19389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:52.339111 19394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:52.339147 19395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:52.339066 19397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:52.340130 19389 server_base.cc:1034] running on GCE node
I20260620 20:12:52.340879 19389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:52.341882 19389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:52.343128 19389 hybrid_clock.cc:648] HybridClock initialized: now 1781986372343068 us; error 49 us; skew 500 ppm
I20260620 20:12:52.343494 19389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:52 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986372, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:52.346863 19389 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:12:52.348460 19389 webserver.cc:460] Webserver started at http://127.4.169.130:39949/ using document root <none> and password file <none>
I20260620 20:12:52.349149 19389 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:52.349294 19389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:52.354954 19389 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 20:12:52.357868 19404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:52.359504 19389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:12:52.359694 19389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "4061fba506d04bd2ad541bf3563d88cd"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "ce690fcaf504f45c28d264f7e324daac"
server_key_iv: "e7529f69f712a34abc7084706881c9b0"
server_key_version: "encryptionkey@0"
I20260620 20:12:52.360733 19389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:52.403944 19389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:52.408560 19389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:52.408864 19389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:52.409917 19389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:52.411386 19389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:52.411518 19389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:52.411664 19389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:52.411755 19389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:52.447099 19389 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:42007
I20260620 20:12:52.447192 19516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:42007 every 8 connection(s)
I20260620 20:12:52.448650 19389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:12:52.451525 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 19389
I20260620 20:12:52.452280 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:38967
--local_ip_for_outbound_sockets=127.4.169.131
--tserver_master_addrs=127.4.169.190:45777
--webserver_port=33215
--webserver_interface=127.4.169.131
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:12:52 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986372, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:52.471515 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.451686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:59315 (local address 127.4.169.190:45777)
0620 20:12:52.451887 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:52.451892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:52.453611 (+ 1719us) server_negotiation.cc:407] Connection header received
0620 20:12:52.455640 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:52.455648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:52.455749 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:52.455863 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:52.457634 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.458503 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.460077 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.460677 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.460825 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.465189 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:52.465212 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:52.465217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:52.465262 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:52.468228 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:52.469031 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:52.469037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:52.469041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:52.469179 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:52.469616 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:52.469622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:52.469626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:52.469915 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:52.470074 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:52.470890 (+ 816us) server_negotiation.cc:299] Negotiation successful
0620 20:12:52.471135 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:12:52.472757 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.452228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:59315)
0620 20:12:52.453438 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:52.453482 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:52.455341 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:52.456121 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:52.456130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:52.456735 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:52.457449 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:52.457469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.458671 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.458675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:52.459915 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:52.459927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.460936 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.460943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:52.461243 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.462035 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:52.462065 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:52.464867 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:52.468400 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:52.468410 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:52.468428 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:52.468895 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:52.469292 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:52.469297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:52.469301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:52.469503 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:52.470065 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:52.470073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:52.470505 (+ 432us) client_negotiation.cc:769] Sending connection context
0620 20:12:52.470790 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 20:12:52.471357 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":721,"threads_started":1}
I20260620 20:12:52.474357 19517 heartbeater.cc:347] Connected to a master server at 127.4.169.190:45777
I20260620 20:12:52.474749 19517 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:52.475498 19517 heartbeater.cc:511] Master 127.4.169.190:45777 requested a full tablet report, sending...
I20260620 20:12:52.476977 19199 ts_manager.cc:194] Registered new tserver with Master: 4061fba506d04bd2ad541bf3563d88cd (127.4.169.130:42007)
I20260620 20:12:52.477852 19199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:59315
W20260620 20:12:52.800062 19520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:52.800441 19520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:52.800552 19520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:52.808243 19520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:52.808359 19520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:52.808475 19520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:12:52.818569 19520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:38967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=33215
--tserver_master_addrs=127.4.169.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:52.819892 19520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:52.821271 19520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:52.835162 19528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:52.835400 19526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:52.835403 19525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:52.836076 19520 server_base.cc:1034] running on GCE node
I20260620 20:12:52.836663 19520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:52.837579 19520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:52.838781 19520 hybrid_clock.cc:648] HybridClock initialized: now 1781986372838725 us; error 40 us; skew 500 ppm
I20260620 20:12:52.839099 19520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:52 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986372, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:52.842373 19520 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:12:52.843984 19520 webserver.cc:460] Webserver started at http://127.4.169.131:33215/ using document root <none> and password file <none>
I20260620 20:12:52.844712 19520 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:52.844879 19520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:52.850483 19520 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:12:52.853327 19535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:52.854992 19520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:12:52.855141 19520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "fd3bab861b0d47f8a849d71c0d61041a"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "95fc1e229785c64f1233a3921fb5277d"
server_key_iv: "a10f20d3119b2b5bdd819d3346c00d39"
server_key_version: "encryptionkey@0"
I20260620 20:12:52.856127 19520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:52.871613 19520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:52.876034 19520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:52.876271 19520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:52.877223 19520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:52.878661 19520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:52.878749 19520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:52.878820 19520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:52.878878 19520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:52.913817 19520 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:38967
I20260620 20:12:52.913906 19647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:38967 every 8 connection(s)
I20260620 20:12:52.915468 19520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:12:52.922207 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 19520
Jun 20 20:12:52 dist-test-slave-4bwq krb5kdc[18613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986372, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:12:52.938802 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.923163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.4.169.190:45777)
0620 20:12:52.923942 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:52.923951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:52.923971 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:12:52.924022 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:52.924030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:52.924093 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:52.924203 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:52.925870 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.926785 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.928309 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.928774 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.928915 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.931842 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:52.931848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:52.931852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:52.931930 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:52.935736 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:52.936599 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:52.936602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:52.936604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:52.936705 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:52.937090 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:52.937095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:52.937098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:52.937406 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:52.937605 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:52.938031 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 20:12:52.938339 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":569,"threads_started":1}
I20260620 20:12:52.938846 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.918205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:34167 (local address 127.4.169.190:45777)
0620 20:12:52.918488 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:52.918493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:52.919712 (+ 1219us) server_negotiation.cc:407] Connection header received
0620 20:12:52.921582 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:52.921588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:52.921686 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:52.921837 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:52.923948 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.924715 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.926553 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.927026 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.927158 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.930908 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:52.930913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:52.930916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:52.930947 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:52.934990 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:52.936141 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:52.936145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:52.936148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:52.936265 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:52.936782 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:52.936789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:52.936795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:52.937149 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:52.937295 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:52.938210 (+ 915us) server_negotiation.cc:299] Negotiation successful
0620 20:12:52.938540 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":89}
I20260620 20:12:52.940086 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.918831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:34167)
0620 20:12:52.919535 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:52.919580 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:52.921291 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:52.922104 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:52.922113 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:52.922797 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:12:52.923681 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:52.923716 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.924882 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.924888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:52.926376 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:52.926395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.927263 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.927270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:52.927530 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.928335 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:12:52.928358 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:12:52.930664 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:12:52.935166 (+ 4502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:52.935175 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:52.935190 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:52.936005 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:52.936383 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:12:52.936388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:12:52.936394 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:12:52.936666 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:12:52.937277 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:12:52.937283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:12:52.937768 (+ 485us) client_negotiation.cc:769] Sending connection context
0620 20:12:52.938085 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 20:12:52.938687 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":270,"threads_started":1}
I20260620 20:12:52.941702 19648 heartbeater.cc:347] Connected to a master server at 127.4.169.190:45777
I20260620 20:12:52.942024 19648 heartbeater.cc:464] Registering TS with master...
I20260620 20:12:52.942783 19648 heartbeater.cc:511] Master 127.4.169.190:45777 requested a full tablet report, sending...
I20260620 20:12:52.944170 19199 ts_manager.cc:194] Registered new tserver with Master: fd3bab861b0d47f8a849d71c0d61041a (127.4.169.131:38967)
I20260620 20:12:52.945039 19199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:34167
I20260620 20:12:52.954074 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:12:52.962152 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.955170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.4.169.190:45777)
0620 20:12:52.955403 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:52.955409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:52.955913 (+ 504us) server_negotiation.cc:407] Connection header received
0620 20:12:52.956092 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:52.956097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:52.956266 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:52.956430 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:12:52.956867 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.958007 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.959797 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.960404 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.960508 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.961183 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:12:52.961650 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:12:52.961761 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:52.961894 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 20:12:52.961966 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:12:52.963161 19199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:12:52.970995 19199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:12:52.996173 19385 heartbeater.cc:503] Master 127.4.169.190:45777 was elected leader, sending a full tablet report...
I20260620 20:12:52.996496 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.984358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52406 (local address 127.4.169.131:38967)
0620 20:12:52.985511 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:52.985525 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:52.987013 (+ 1488us) server_negotiation.cc:407] Connection header received
0620 20:12:52.987320 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:52.987330 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:52.987597 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:52.987770 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:52.988793 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.990471 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.993389 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.994885 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.994987 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.995244 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:52.995938 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 20:12:52.996124 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":793,"threads_started":1}
I20260620 20:12:52.997387 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.983853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:38967 (local address 127.0.0.1:52406)
0620 20:12:52.986901 (+ 3048us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:52.986919 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:52.987114 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:52.988129 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:52.988137 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:52.988168 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:52.988599 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:52.988614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.990649 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.990654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:52.993209 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:52.993225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.995435 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.995440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:52.995645 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.995700 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 20:12:52.996741 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0620 20:12:52.996967 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2912,"mutex_wait_us":4,"thread_start_us":1879,"threads_started":1}
I20260620 20:12:52.998541 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.983817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:37100)
0620 20:12:52.986138 (+ 2321us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:52.986175 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:52.986343 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:52.988774 (+ 2431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:52.988778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:52.988801 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:52.989198 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:52.989213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.992470 (+ 3257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.992482 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:52.994614 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:52.994629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.997734 (+ 3105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.997740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:52.997937 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.997975 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:12:52.998044 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 20:12:52.998187 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2083,"thread_start_us":2792,"threads_started":1}
I20260620 20:12:53.000252 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.984397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.4.169.129:42943)
0620 20:12:52.988180 (+ 3783us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:52.988191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:52.988222 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:12:52.988319 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:52.988325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:52.988565 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:52.989422 (+ 857us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:52.989650 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.992300 (+ 2650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.995938 (+ 3638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:52.997467 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:52.999060 (+ 1593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:52.999312 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:52.999394 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 20:12:52.999563 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2458,"thread_start_us":2294,"threads_started":1}
I20260620 20:12:53.002166 19583 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3455c5385242cfac7316bfd8e7c7a6 (DEFAULT_TABLE table=test-table [id=7815284055bd4d7fbf053b10f305d53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:12:53.003592 19583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c3455c5385242cfac7316bfd8e7c7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:53.005264 19320 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3455c5385242cfac7316bfd8e7c7a6 (DEFAULT_TABLE table=test-table [id=7815284055bd4d7fbf053b10f305d53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:12:53.006646 19320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c3455c5385242cfac7316bfd8e7c7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:53.008908 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.987049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:42007 (local address 127.0.0.1:54292)
0620 20:12:52.998773 (+ 11724us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:52.998792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:52.998890 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:53.002128 (+ 3238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:53.002132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:53.002160 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:53.002347 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:53.002358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.004300 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.004305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:53.006124 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:53.006142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.008088 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.008094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:53.008283 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.008323 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 20:12:53.008428 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:12:53.008601 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11625,"thread_start_us":14434,"threads_started":1}
I20260620 20:12:53.009011 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:52.987306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.4.169.130:42007)
0620 20:12:53.001504 (+ 14198us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:53.001513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:53.001544 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:12:53.001646 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:53.001654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:53.001893 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:53.002142 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:53.002541 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.004149 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.006332 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.007893 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.008039 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.008244 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:53.008485 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 20:12:53.008669 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13957,"thread_start_us":13795,"threads_started":1}
I20260620 20:12:53.013342 19452 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3455c5385242cfac7316bfd8e7c7a6 (DEFAULT_TABLE table=test-table [id=7815284055bd4d7fbf053b10f305d53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:12:53.014891 19452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c3455c5385242cfac7316bfd8e7c7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:12:53.019794 19665 tablet_bootstrap.cc:492] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: Bootstrap starting.
I20260620 20:12:53.020588 19664 tablet_bootstrap.cc:492] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: Bootstrap starting.
I20260620 20:12:53.024096 19664 tablet_bootstrap.cc:654] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:53.025447 19664 log.cc:824] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:53.026000 19665 tablet_bootstrap.cc:654] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:53.027235 19665 log.cc:824] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:53.029451 19664 tablet_bootstrap.cc:492] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: No bootstrap required, opened a new log
I20260620 20:12:53.029757 19664 ts_tablet_manager.cc:1397] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:12:53.036262 19668 tablet_bootstrap.cc:492] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: Bootstrap starting.
I20260620 20:12:53.036306 19665 tablet_bootstrap.cc:492] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: No bootstrap required, opened a new log
I20260620 20:12:53.036957 19665 ts_tablet_manager.cc:1397] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20260620 20:12:53.040131 19668 tablet_bootstrap.cc:654] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: Neither blocks nor log segments found. Creating new log.
I20260620 20:12:53.041409 19668 log.cc:824] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:53.041692 19664 raft_consensus.cc:348] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } }
I20260620 20:12:53.042208 19664 raft_consensus.cc:374] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:53.042336 19664 raft_consensus.cc:729] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e529e4c6b85e4cd19db1b5adf6a5f9a2, State: Initialized, Role: FOLLOWER
I20260620 20:12:53.043030 19664 consensus_queue.cc:260] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } }
I20260620 20:12:53.044824 19668 tablet_bootstrap.cc:492] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: No bootstrap required, opened a new log
I20260620 20:12:53.045104 19668 ts_tablet_manager.cc:1397] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:12:53.045080 19665 raft_consensus.cc:348] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } }
I20260620 20:12:53.045507 19665 raft_consensus.cc:374] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:53.045614 19665 raft_consensus.cc:729] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3bab861b0d47f8a849d71c0d61041a, State: Initialized, Role: FOLLOWER
I20260620 20:12:53.046120 19664 ts_tablet_manager.cc:1428] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260620 20:12:53.046250 19665 consensus_queue.cc:260] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } }
I20260620 20:12:53.047448 19648 heartbeater.cc:503] Master 127.4.169.190:45777 was elected leader, sending a full tablet report...
I20260620 20:12:53.048338 19665 ts_tablet_manager.cc:1428] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 20:12:53.054808 19668 raft_consensus.cc:348] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } }
I20260620 20:12:53.055222 19668 raft_consensus.cc:374] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:12:53.055342 19668 raft_consensus.cc:729] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4061fba506d04bd2ad541bf3563d88cd, State: Initialized, Role: FOLLOWER
I20260620 20:12:53.055925 19668 consensus_queue.cc:260] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } }
I20260620 20:12:53.057036 19517 heartbeater.cc:503] Master 127.4.169.190:45777 was elected leader, sending a full tablet report...
I20260620 20:12:53.057502 19668 ts_tablet_manager.cc:1428] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260620 20:12:53.137535 19672 raft_consensus.cc:482] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:12:53.137817 19672 raft_consensus.cc:504] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } }
I20260620 20:12:53.139911 19672 leader_election.cc:288] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd3bab861b0d47f8a849d71c0d61041a (127.4.169.131:38967), e529e4c6b85e4cd19db1b5adf6a5f9a2 (127.4.169.129:42943)
I20260620 20:12:53.149504 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.140287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:49887 (local address 127.4.169.131:38967)
0620 20:12:53.140484 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:53.140490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:53.141384 (+ 894us) server_negotiation.cc:407] Connection header received
0620 20:12:53.141484 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:53.141488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:53.141566 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:53.141653 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:53.142644 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.143769 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.147031 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.148291 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.148469 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.148547 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:53.149100 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 20:12:53.149238 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:12:53.149504 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.140495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:33517 (local address 127.4.169.129:42943)
0620 20:12:53.140718 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:53.140724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:53.141214 (+ 490us) server_negotiation.cc:407] Connection header received
0620 20:12:53.141434 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:53.141440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:53.141525 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:53.141632 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:12:53.142644 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.143766 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.147031 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.148277 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.148468 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.148547 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:53.149100 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 20:12:53.149238 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:12:53.149784 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.140246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:38967 (local address 127.4.169.130:49887)
0620 20:12:53.141244 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.141259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:53.141365 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:53.141675 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:53.141680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:53.141770 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:53.142417 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:53.142440 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.143923 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.143929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:53.146823 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:53.146841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.148619 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.148624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:53.148888 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.148947 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:12:53.149048 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:12:53.149250 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":320,"spinlock_wait_cycles":24832,"thread_start_us":422,"threads_started":1}
I20260620 20:12:53.150085 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.140181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.4.169.130:33517)
0620 20:12:53.141085 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.141107 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:12:53.141267 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:12:53.141675 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:12:53.141680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:12:53.141709 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:12:53.142493 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:53.142505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.143923 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.143929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:53.146823 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:12:53.146841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.148616 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.148623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:12:53.148921 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.148950 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:12:53.149048 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:12:53.149250 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":419,"thread_start_us":735,"threads_started":1}
I20260620 20:12:53.150944 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3455c5385242cfac7316bfd8e7c7a6" candidate_uuid: "4061fba506d04bd2ad541bf3563d88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3bab861b0d47f8a849d71c0d61041a" is_pre_election: true
I20260620 20:12:53.151072 19340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3455c5385242cfac7316bfd8e7c7a6" candidate_uuid: "4061fba506d04bd2ad541bf3563d88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" is_pre_election: true
I20260620 20:12:53.151464 19603 raft_consensus.cc:2454] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4061fba506d04bd2ad541bf3563d88cd in term 0.
I20260620 20:12:53.151602 19340 raft_consensus.cc:2454] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4061fba506d04bd2ad541bf3563d88cd in term 0.
I20260620 20:12:53.152132 19405 leader_election.cc:302] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4061fba506d04bd2ad541bf3563d88cd, fd3bab861b0d47f8a849d71c0d61041a; no voters:
I20260620 20:12:53.152498 19672 raft_consensus.cc:2781] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:12:53.152645 19672 raft_consensus.cc:482] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:12:53.152753 19672 raft_consensus.cc:3037] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:53.155063 19672 raft_consensus.cc:504] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } }
I20260620 20:12:53.155656 19672 leader_election.cc:288] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [CANDIDATE]: Term 1 election: Requested vote from peers fd3bab861b0d47f8a849d71c0d61041a (127.4.169.131:38967), e529e4c6b85e4cd19db1b5adf6a5f9a2 (127.4.169.129:42943)
I20260620 20:12:53.156102 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3455c5385242cfac7316bfd8e7c7a6" candidate_uuid: "4061fba506d04bd2ad541bf3563d88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3bab861b0d47f8a849d71c0d61041a"
I20260620 20:12:53.156170 19340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3455c5385242cfac7316bfd8e7c7a6" candidate_uuid: "4061fba506d04bd2ad541bf3563d88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2"
I20260620 20:12:53.156297 19603 raft_consensus.cc:3037] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:53.156435 19340 raft_consensus.cc:3037] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:12:53.158738 19603 raft_consensus.cc:2454] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4061fba506d04bd2ad541bf3563d88cd in term 1.
I20260620 20:12:53.158799 19340 raft_consensus.cc:2454] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4061fba506d04bd2ad541bf3563d88cd in term 1.
I20260620 20:12:53.159242 19407 leader_election.cc:302] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4061fba506d04bd2ad541bf3563d88cd, e529e4c6b85e4cd19db1b5adf6a5f9a2, fd3bab861b0d47f8a849d71c0d61041a; no voters:
I20260620 20:12:53.159608 19672 raft_consensus.cc:2781] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:12:53.160113 19672 raft_consensus.cc:686] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 1 LEADER]: Becoming Leader. State: Replica: 4061fba506d04bd2ad541bf3563d88cd, State: Running, Role: LEADER
I20260620 20:12:53.160701 19672 consensus_queue.cc:237] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } }
I20260620 20:12:53.165176 19199 catalog_manager.cc:5495] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4061fba506d04bd2ad541bf3563d88cd (127.4.169.130). New cstate: current_term: 1 leader_uuid: "4061fba506d04bd2ad541bf3563d88cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4061fba506d04bd2ad541bf3563d88cd" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 42007 } health_report { overall_health: HEALTHY } } }
W20260620 20:12:53.167124 19649 tablet.cc:2369] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:12:53.188117 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.182859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.4.169.130:42007)
0620 20:12:53.183041 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:53.183046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:53.183063 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:12:53.183157 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:53.183161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:53.183247 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:53.183377 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:12:53.183757 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.184480 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.185849 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.186318 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.186484 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.187086 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:12:53.187591 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:12:53.187711 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:53.187795 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 20:12:53.187864 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:12:53.196419 19603 raft_consensus.cc:1261] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [term 1 FOLLOWER]: Refusing update from remote peer 4061fba506d04bd2ad541bf3563d88cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:12:53.196431 19340 raft_consensus.cc:1261] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 1 FOLLOWER]: Refusing update from remote peer 4061fba506d04bd2ad541bf3563d88cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:12:53.197418 19675 consensus_queue.cc:1035] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3bab861b0d47f8a849d71c0d61041a" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:12:53.197782 19672 consensus_queue.cc:1035] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:12:53.200667 19518 tablet.cc:2369] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:12:53.208748 19386 tablet.cc:2369] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:12:53.224390 19199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:12:53.224624 19199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:12:53.228233 19199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Sending DeleteTablet for 3 replicas of tablet 4c3455c5385242cfac7316bfd8e7c7a6
I20260620 20:12:53.229499 19583 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3455c5385242cfac7316bfd8e7c7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:12:53 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:52406
I20260620 20:12:53.229532 19452 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3455c5385242cfac7316bfd8e7c7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:12:53 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:54292
I20260620 20:12:53.229774 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 19236
I20260620 20:12:53.230705 19688 tablet_replica.cc:330] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: stopping tablet replica
I20260620 20:12:53.231130 19688 raft_consensus.cc:2229] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:12:53.231331 19320 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3455c5385242cfac7316bfd8e7c7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:12:53 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:37100
I20260620 20:12:53.231776 19688 raft_consensus.cc:2258] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:12:53.232508 19687 tablet_replica.cc:330] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: stopping tablet replica
I20260620 20:12:53.232908 19687 raft_consensus.cc:2229] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:12:53.233390 19687 raft_consensus.cc:2258] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:12:53.235491 19688 ts_tablet_manager.cc:1905] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:12:53.242977 19688 ts_tablet_manager.cc:1918] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:12:53.243178 19688 log.cc:1196] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/4c3455c5385242cfac7316bfd8e7c7a6
I20260620 20:12:53.243788 19688 ts_tablet_manager.cc:1939] T 4c3455c5385242cfac7316bfd8e7c7a6 P 4061fba506d04bd2ad541bf3563d88cd: Deleting consensus metadata
I20260620 20:12:53.245714 19185 catalog_manager.cc:4841] TS 4061fba506d04bd2ad541bf3563d88cd (127.4.169.130:42007): tablet 4c3455c5385242cfac7316bfd8e7c7a6 (table test-table [id=7815284055bd4d7fbf053b10f305d53a]) successfully deleted
I20260620 20:12:53.246268 19689 tablet_replica.cc:330] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: stopping tablet replica
I20260620 20:12:53.246794 19689 raft_consensus.cc:2229] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:12:53.247256 19687 ts_tablet_manager.cc:1905] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:12:53.247361 19689 raft_consensus.cc:2258] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:12:53.249382 19689 ts_tablet_manager.cc:1905] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:12:53.254173 19687 ts_tablet_manager.cc:1918] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:12:53.254355 19687 log.cc:1196] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/4c3455c5385242cfac7316bfd8e7c7a6
I20260620 20:12:53.254935 19687 ts_tablet_manager.cc:1939] T 4c3455c5385242cfac7316bfd8e7c7a6 P fd3bab861b0d47f8a849d71c0d61041a: Deleting consensus metadata
I20260620 20:12:53.255093 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.230948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.4.169.129:42943)
0620 20:12:53.231149 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:53.231158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:53.232170 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 20:12:53.232373 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:53.232379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:53.232475 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:53.232575 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:53.234358 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.236027 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.237347 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.238272 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.238428 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.242748 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:53.242775 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:53.242803 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:53.242866 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:53.248970 (+ 6104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:53.251324 (+ 2354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:53.251332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:53.251346 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:53.251496 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:53.251962 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:53.251967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:53.251971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:53.252457 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:53.252767 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:53.254112 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0620 20:12:53.254629 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:12:53.256009 19689 ts_tablet_manager.cc:1918] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:12:53.256117 19380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:53.256196 19689 log.cc:1196] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/4c3455c5385242cfac7316bfd8e7c7a6
I20260620 20:12:53.256920 19183 catalog_manager.cc:4841] TS fd3bab861b0d47f8a849d71c0d61041a (127.4.169.131:38967): tablet 4c3455c5385242cfac7316bfd8e7c7a6 (table test-table [id=7815284055bd4d7fbf053b10f305d53a]) successfully deleted
I20260620 20:12:53.349710 19689 ts_tablet_manager.cc:1939] T 4c3455c5385242cfac7316bfd8e7c7a6 P e529e4c6b85e4cd19db1b5adf6a5f9a2: Deleting consensus metadata
I20260620 20:12:53.350730 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 19236
W20260620 20:12:53.370225 19185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.169.129:42943 (error 108)
I20260620 20:12:53.370496 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 19389
I20260620 20:12:53.372476 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.371861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:37124)
0620 20:12:53.372080 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.372257 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 20:12:53.372764 19185 catalog_manager.cc:4584] TS e529e4c6b85e4cd19db1b5adf6a5f9a2 (127.4.169.129:42943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c3455c5385242cfac7316bfd8e7c7a6: Network error: Client connection negotiation failed: client connection to 127.4.169.129:42943: connect: Connection refused (error 111)
I20260620 20:12:53.398221 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.371415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.4.169.130:42007)
0620 20:12:53.371630 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:53.371637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:53.372232 (+ 595us) server_negotiation.cc:407] Connection header received
0620 20:12:53.372452 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:53.372461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:53.372560 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:53.372678 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:53.375010 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.375873 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.377307 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.377767 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.377911 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.391406 (+ 13495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:53.391447 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:53.391470 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:53.391528 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:53.394973 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:53.395870 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:53.395882 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:53.395885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:53.396055 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:53.396482 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:53.396489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:53.396492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:53.396879 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:53.397248 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:53.397601 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 20:12:53.397836 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:12:53.399008 19512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:53.426880 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.426305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:37128)
0620 20:12:53.426574 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.426686 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 20:12:53.428023 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.427589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:37132)
0620 20:12:53.427783 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.427873 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 20:12:53.488595 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.488117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:37148)
0620 20:12:53.488342 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.488441 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 20:12:53.489624 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.489247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:37156)
0620 20:12:53.489415 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.489482 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:12:53.517895 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 19389
I20260620 20:12:53.537717 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 19520
I20260620 20:12:53.552222 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.538568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.4.169.131:38967)
0620 20:12:53.538781 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:12:53.538788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:12:53.538934 (+ 146us) server_negotiation.cc:407] Connection header received
0620 20:12:53.539154 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:12:53.539160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:12:53.539234 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:12:53.539366 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:12:53.540737 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.541474 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.543100 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:12:53.543511 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:12:53.543648 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:12:53.546375 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:12:53.546436 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:12:53.546462 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:12:53.546504 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:12:53.549275 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:53.550046 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:53.550062 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:53.550064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:53.550211 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:12:53.550662 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:12:53.550669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:12:53.550672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:12:53.551038 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:12:53.551345 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 20:12:53.551669 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 20:12:53.551886 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:12:53.552932 19643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:53.571126 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.570614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:37160)
0620 20:12:53.570880 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.570977 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 20:12:53.572237 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.571854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:37170)
0620 20:12:53.572045 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.572122 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 20:12:53.645617 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 19520
I20260620 20:12:53.662930 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 19166
I20260620 20:12:53.664163 19229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:12:53.780438 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:53.779759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:42943 (local address 127.0.0.1:37184)
0620 20:12:53.780027 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:53.780173 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 20:12:53.780663 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 19166
I20260620 20:12:53.801990 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45777
--webserver_interface=127.4.169.190
--webserver_port=45187
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:45777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:54.136402 19695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:54.136700 19695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:54.136791 19695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:54.144177 19695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:12:54.144307 19695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:54.144356 19695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:54.144408 19695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:12:54.144454 19695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:12:54.154589 19695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:45777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=45187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:54.155854 19695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:54.157263 19695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:54.169021 19701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:54.169121 19703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:54.169128 19700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:54.169849 19695 server_base.cc:1034] running on GCE node
I20260620 20:12:54.170689 19695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:54.172148 19695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:54.173393 19695 hybrid_clock.cc:648] HybridClock initialized: now 1781986374173342 us; error 48 us; skew 500 ppm
I20260620 20:12:54.173745 19695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:54 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986374, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:54.176893 19695 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:12:54.178462 19695 webserver.cc:460] Webserver started at http://127.4.169.190:45187/ using document root <none> and password file <none>
I20260620 20:12:54.179183 19695 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:54.179332 19695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:54.184906 19695 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 20:12:54.187983 19710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:54.189359 19695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:12:54.189549 19695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "3c79daeeed124df5821f34eb1c131e50"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "44dfb58d313334df366e9cff04633d72"
server_key_iv: "24d299450aa9412016e869c429a55fc5"
server_key_version: "encryptionkey@0"
I20260620 20:12:54.190618 19695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:54.202870 19695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:54.206857 19695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:54.207137 19695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:54.226186 19695 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:45777
I20260620 20:12:54.226207 19761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:45777 every 8 connection(s)
I20260620 20:12:54.227854 19695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:12:54.235574 19762 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:12:54.235832 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 19695
I20260620 20:12:54.236836 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:42943
--local_ip_for_outbound_sockets=127.4.169.129
--tserver_master_addrs=127.4.169.190:45777
--webserver_port=39187
--webserver_interface=127.4.169.129
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:12:54.240731 19762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Bootstrap starting.
I20260620 20:12:54.264081 19762 log.cc:824] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Log is configured to *not* fsync() on all Append() calls
I20260620 20:12:54.280064 19762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:12:54.280615 19762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50: Bootstrap complete.
I20260620 20:12:54.289657 19762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:54.290601 19762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c79daeeed124df5821f34eb1c131e50, State: Initialized, Role: FOLLOWER
I20260620 20:12:54.291261 19762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:54.291487 19762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:12:54.291604 19762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:12:54.291788 19762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:12:54.295923 19762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:54.296437 19762 leader_election.cc:302] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c79daeeed124df5821f34eb1c131e50; no voters:
I20260620 20:12:54.297103 19762 leader_election.cc:288] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:12:54.297240 19766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:12:54.298679 19766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [term 3 LEADER]: Becoming Leader. State: Replica: 3c79daeeed124df5821f34eb1c131e50, State: Running, Role: LEADER
I20260620 20:12:54.299458 19762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:12:54.299292 19766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } }
I20260620 20:12:54.302003 19768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c79daeeed124df5821f34eb1c131e50. Latest consensus state: current_term: 3 leader_uuid: "3c79daeeed124df5821f34eb1c131e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } } }
I20260620 20:12:54.302496 19768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:54.303121 19767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3c79daeeed124df5821f34eb1c131e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c79daeeed124df5821f34eb1c131e50" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45777 } } }
I20260620 20:12:54.303428 19767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c79daeeed124df5821f34eb1c131e50 [sys.catalog]: This master's current role is: LEADER
I20260620 20:12:54.305670 19772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:12:54.316854 19772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:12:54.318370 19772 catalog_manager.cc:1259] Loaded cluster ID: 03e2feaa6b9b455d8040ab95c71fe8f3
I20260620 20:12:54.318550 19772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 20:12:54.320633 19772 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb6757419ad google::LogMessage::Fail() at ??:0
@ 0x7fb67574488a google::LogMessage::SendToLog() at ??:0
@ 0x7fb6757414de google::LogMessage::Flush() at ??:0
@ 0x7fb675742a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb68b178512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb68b175976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb68b1fd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb68b1fd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb67838d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb67839091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb678390799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb67836beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb67b6456db start_thread at ??:0
@ 0x7fb6745db61f clone at ??:0
W20260620 20:12:54.621168 19764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:54.621548 19764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:54.621727 19764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:54.629390 19764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:54.629566 19764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:54.629761 19764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:12:54.640445 19764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:42943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=39187
--tserver_master_addrs=127.4.169.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:54.641906 19764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:54.643365 19764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:54.656669 19802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:54.656693 19805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:54.656622 19803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:54.657912 19764 server_base.cc:1034] running on GCE node
I20260620 20:12:54.658668 19764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:54.659727 19764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:54.661000 19764 hybrid_clock.cc:648] HybridClock initialized: now 1781986374660954 us; error 41 us; skew 500 ppm
I20260620 20:12:54.661350 19764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:54 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986374, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:54.664606 19764 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:12:54.666038 19764 webserver.cc:460] Webserver started at http://127.4.169.129:39187/ using document root <none> and password file <none>
I20260620 20:12:54.666824 19764 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:54.666975 19764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:54.672323 19764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:12:54.676898 19812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:54.678267 19764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:12:54.678484 19764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "e529e4c6b85e4cd19db1b5adf6a5f9a2"
format_stamp: "Formatted at 2026-06-20 20:12:48 on dist-test-slave-4bwq"
server_key: "c5f9f3abac5f9da993ffa20959dfef03"
server_key_iv: "8d8ee657d97168becaa7d0b976801915"
server_key_version: "encryptionkey@0"
I20260620 20:12:54.679523 19764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:54.704923 19764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:54.709143 19764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:54.709437 19764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:54.710580 19764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:54.711970 19764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:54.712100 19764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:54.712230 19764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:54.712321 19764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:54.746862 19764 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:42943
I20260620 20:12:54.746951 19924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:42943 every 8 connection(s)
I20260620 20:12:54.748376 19764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:12:54.753176 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:54.751367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:60335)
0620 20:12:54.752075 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:54.752341 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":261,"threads_started":1}
W20260620 20:12:54.753948 19925 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:45777 (0 consecutive failures): Network error: Failed to ping master at 127.4.169.190:45777: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
I20260620 20:12:54.754204 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 19764
I20260620 20:12:54.755004 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:54.754576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:42689)
0620 20:12:54.754762 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:54.754848 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 20:12:54.755060 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:42007
--local_ip_for_outbound_sockets=127.4.169.130
--tserver_master_addrs=127.4.169.190:45777
--webserver_port=39949
--webserver_interface=127.4.169.130
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:12:54.755956 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:54.755524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:36115)
0620 20:12:54.755695 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:54.755797 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:12:54.756201 19925 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:12:55.088001 19928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:55.088469 19928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:55.088704 19928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:55.098985 19928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:55.099136 19928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:55.099336 19928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:12:55.109424 19928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:42007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=39949
--tserver_master_addrs=127.4.169.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:55.110893 19928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:55.112243 19928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:55.125864 19933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:55.125950 19934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:55.125974 19936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:55.126643 19928 server_base.cc:1034] running on GCE node
I20260620 20:12:55.127290 19928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:55.128401 19928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:55.129669 19928 hybrid_clock.cc:648] HybridClock initialized: now 1781986375129618 us; error 46 us; skew 500 ppm
I20260620 20:12:55.130062 19928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:55 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986375, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:55.133502 19928 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:12:55.135073 19928 webserver.cc:460] Webserver started at http://127.4.169.130:39949/ using document root <none> and password file <none>
I20260620 20:12:55.135768 19928 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:55.135877 19928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:55.141367 19928 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:12:55.144332 19943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:55.145860 19928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:12:55.146045 19928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "4061fba506d04bd2ad541bf3563d88cd"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "ce690fcaf504f45c28d264f7e324daac"
server_key_iv: "e7529f69f712a34abc7084706881c9b0"
server_key_version: "encryptionkey@0"
I20260620 20:12:55.147234 19928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:55.166496 19928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:55.170799 19928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:55.171092 19928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:55.172056 19928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:55.173357 19928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:55.173441 19928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:55.173612 19928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:55.173697 19928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:55.207311 19928 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:42007
I20260620 20:12:55.207378 20055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:42007 every 8 connection(s)
I20260620 20:12:55.208845 19928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:12:55.214087 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:55.212113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:33063)
0620 20:12:55.213001 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:55.213236 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":377,"threads_started":1}
W20260620 20:12:55.214885 20056 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:45777 (0 consecutive failures): Network error: Failed to ping master at 127.4.169.190:45777: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
I20260620 20:12:55.215739 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:55.215425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:42359)
0620 20:12:55.215570 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:55.215636 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 20:12:55.216572 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:55.216288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:47955)
0620 20:12:55.216402 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:55.216472 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:12:55.216849 20056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:12:55.218159 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 19928
I20260620 20:12:55.218844 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:38967
--local_ip_for_outbound_sockets=127.4.169.131
--tserver_master_addrs=127.4.169.190:45777
--webserver_port=33215
--webserver_interface=127.4.169.131
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:12:55.562657 20059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:12:55.562951 20059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:12:55.563146 20059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:12:55.570566 20059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:12:55.570676 20059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:12:55.570860 20059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:12:55.580708 20059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:38967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=33215
--tserver_master_addrs=127.4.169.190:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:12:55.582070 20059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:12:55.583591 20059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:12:55.596861 20065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:55.596938 20064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:12:55.596938 20067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:12:55.597733 20059 server_base.cc:1034] running on GCE node
I20260620 20:12:55.598311 20059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:12:55.599283 20059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:12:55.600474 20059 hybrid_clock.cc:648] HybridClock initialized: now 1781986375600431 us; error 40 us; skew 500 ppm
I20260620 20:12:55.600744 20059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:12:55 dist-test-slave-4bwq krb5kdc[18613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986375, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:12:55.603930 20059 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:12:55.605276 20059 webserver.cc:460] Webserver started at http://127.4.169.131:33215/ using document root <none> and password file <none>
I20260620 20:12:55.605921 20059 fs_manager.cc:359] Metadata directory not provided
I20260620 20:12:55.606017 20059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:12:55.611202 20059 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:12:55.613785 20074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:55.615276 20059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:12:55.615460 20059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "fd3bab861b0d47f8a849d71c0d61041a"
format_stamp: "Formatted at 2026-06-20 20:12:49 on dist-test-slave-4bwq"
server_key: "95fc1e229785c64f1233a3921fb5277d"
server_key_iv: "a10f20d3119b2b5bdd819d3346c00d39"
server_key_version: "encryptionkey@0"
I20260620 20:12:55.616425 20059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:12:55.646713 20059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:12:55.651000 20059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:12:55.651289 20059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:12:55.652302 20059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:12:55.653652 20059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:12:55.653787 20059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:55.654098 20059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:12:55.654325 20059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:12:55.688766 20059 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:38967
I20260620 20:12:55.688859 20186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:38967 every 8 connection(s)
I20260620 20:12:55.690243 20059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:12:55.692543 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 20059
I20260620 20:12:55.695478 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:55.693493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:38617)
0620 20:12:55.694284 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:55.694626 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":329,"threads_started":1}
W20260620 20:12:55.696161 20187 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:45777 (0 consecutive failures): Network error: Failed to ping master at 127.4.169.190:45777: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
I20260620 20:12:55.696990 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:55.696669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:54101)
0620 20:12:55.696818 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:55.696895 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 20:12:55.697867 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:55.697547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:59081)
0620 20:12:55.697682 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:55.697755 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:12:55.698102 20187 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:12:55.757534 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:55.756762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:35743)
0620 20:12:55.757214 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:55.757311 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":222,"threads_started":1}
I20260620 20:12:56.218290 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:56.217420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:48355)
0620 20:12:56.217933 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:56.218038 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":204,"threads_started":1}
I20260620 20:12:56.699728 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:56.698872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:35825)
0620 20:12:56.699391 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:56.699490 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
I20260620 20:12:56.759492 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:56.758657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:53657)
0620 20:12:56.759220 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:56.759318 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":303,"threads_started":1}
I20260620 20:12:57.220062 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:57.219240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:41971)
0620 20:12:57.219799 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:57.219888 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":245,"threads_started":1}
I20260620 20:12:57.701504 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:57.700677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:41159)
0620 20:12:57.701239 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:57.701329 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":308,"threads_started":1}
I20260620 20:12:57.761268 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:57.760545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:37753)
0620 20:12:57.760999 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:57.761089 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260620 20:12:58.221766 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:58.221039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:55099)
0620 20:12:58.221500 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:58.221587 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
I20260620 20:12:58.703219 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:58.702510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:39221)
0620 20:12:58.702962 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:58.703044 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":247,"threads_started":1}
I20260620 20:12:58.763137 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:58.762307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:35325)
0620 20:12:58.762863 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:58.762957 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":290,"threads_started":1}
I20260620 20:12:59.223449 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:59.222685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:51853)
0620 20:12:59.223147 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:59.223229 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":242,"threads_started":1}
I20260620 20:12:59.705019 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:59.704254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:55431)
0620 20:12:59.704746 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:59.704841 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":252,"threads_started":1}
I20260620 20:12:59.764885 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:12:59.764174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:44747)
0620 20:12:59.764631 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 20:12:59.764712 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":227,"threads_started":1}
I20260620 20:13:00.225409 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:00.224577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:55387)
0620 20:13:00.225124 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:00.225218 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":295,"threads_started":1}
I20260620 20:13:00.707062 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:00.706130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:33543)
0620 20:13:00.706749 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:00.706848 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":357,"threads_started":1}
I20260620 20:13:00.766746 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:00.765906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:46623)
0620 20:13:00.766455 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:00.766562 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":267,"threads_started":1}
I20260620 20:13:01.227449 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:01.226661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:40683)
0620 20:13:01.227173 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:01.227272 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":275,"threads_started":1}
I20260620 20:13:01.708828 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:01.708063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:41165)
0620 20:13:01.708545 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:01.708643 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
I20260620 20:13:01.768584 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:01.767713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:54923)
0620 20:13:01.768279 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:01.768375 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":283,"threads_started":1}
I20260620 20:13:02.229573 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:02.228631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:46401)
0620 20:13:02.229194 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:02.229289 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":269,"threads_started":1}
I20260620 20:13:02.710788 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:02.709867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:44307)
0620 20:13:02.710474 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:02.710573 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":241,"threads_started":1}
I20260620 20:13:02.770356 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:02.769592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:38107)
0620 20:13:02.770085 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:02.770183 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":251,"threads_started":1}
I20260620 20:13:03.231441 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:03.230701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:48595)
0620 20:13:03.231170 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:03.231261 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":241,"threads_started":1}
I20260620 20:13:03.712680 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:03.711937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:60911)
0620 20:13:03.712423 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:03.712516 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":242,"threads_started":1}
I20260620 20:13:03.772275 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:03.771475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:43955)
0620 20:13:03.771985 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:03.772080 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":251,"threads_started":1}
I20260620 20:13:04.233369 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:04.232488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:37573)
0620 20:13:04.233093 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:04.233186 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":296,"threads_started":1}
I20260620 20:13:04.714815 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:04.713853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:36411)
0620 20:13:04.714469 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:04.714585 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":300,"threads_started":1}
I20260620 20:13:04.774088 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:04.773328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:38803)
0620 20:13:04.773812 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:04.773914 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":267,"threads_started":1}
I20260620 20:13:05.235110 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:05.234362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:39895)
0620 20:13:05.234857 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:05.234941 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":228,"threads_started":1}
I20260620 20:13:05.716696 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:05.715932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:44917)
0620 20:13:05.716436 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:05.716528 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":275,"threads_started":1}
I20260620 20:13:05.776232 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:05.775352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:58733)
0620 20:13:05.775906 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:05.776034 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":293,"threads_started":1}
I20260620 20:13:06.237030 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:06.236138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:42485)
0620 20:13:06.236704 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:06.236803 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":326,"threads_started":1}
I20260620 20:13:06.718741 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:06.717826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:53243)
0620 20:13:06.718436 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:06.718550 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":270,"threads_started":1}
I20260620 20:13:06.778012 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:06.777281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:58437)
0620 20:13:06.777756 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:06.777853 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":247,"threads_started":1}
I20260620 20:13:07.239053 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:07.238127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:40913)
0620 20:13:07.238765 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:07.238889 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":411,"threads_started":1}
I20260620 20:13:07.720654 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:07.719893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:58981)
0620 20:13:07.720385 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:07.720473 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":251,"threads_started":1}
I20260620 20:13:07.780017 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:07.779180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:51819)
0620 20:13:07.779707 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:07.779818 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":254,"threads_started":1}
I20260620 20:13:08.240947 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:08.240117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:41773)
0620 20:13:08.240625 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:08.240738 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":249,"threads_started":1}
I20260620 20:13:08.722453 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:08.721616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:34281)
0620 20:13:08.722085 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:08.722182 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":235,"threads_started":1}
I20260620 20:13:08.781735 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:08.780971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:57865)
0620 20:13:08.781448 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:08.781546 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":231,"threads_started":1}
I20260620 20:13:09.242718 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:09.241897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:58011)
0620 20:13:09.242358 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:09.242521 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
I20260620 20:13:09.724097 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:09.723372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:51895)
0620 20:13:09.723809 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:09.723908 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":215,"threads_started":1}
I20260620 20:13:09.783815 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:09.783039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.129:46175)
0620 20:13:09.783479 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:09.783596 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260620 20:13:10.244658 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:10.243876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.130:50637)
0620 20:13:10.244356 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:10.244448 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":255,"threads_started":1}
I20260620 20:13:10.694000 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 19764
I20260620 20:13:10.722592 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:10.695175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.4.169.129:42943)
0620 20:13:10.695813 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:10.695832 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:10.695877 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:13:10.698479 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:10.698491 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:10.698936 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:10.699232 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:10.700674 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:10.701994 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:10.703450 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:10.704143 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:10.704284 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:10.715544 (+ 11260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:10.715555 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:10.715575 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:10.715631 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:10.718860 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:10.719640 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:10.719647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:10.719653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:10.719836 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:10.720298 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:10.720304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:10.720307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:10.720898 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:10.721152 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:10.721713 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 20:13:10.722196 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":321,"threads_started":1}
I20260620 20:13:10.724062 19920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:10.729636 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:10.725217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:45777 (local address 127.4.169.131:39779)
0620 20:13:10.729295 (+ 4078us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:10.729413 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:45777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3920,"thread_start_us":3806,"threads_started":1}
I20260620 20:13:10.811192 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 19764
I20260620 20:13:10.828405 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 19928
I20260620 20:13:10.846774 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:10.829484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.4.169.130:42007)
0620 20:13:10.830018 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:10.830033 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:10.830069 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:13:10.831739 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:10.831748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:10.832195 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:10.832565 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:10.834089 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:10.835411 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:10.836729 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:10.837406 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:10.837613 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:10.840318 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:10.840327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:10.840342 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:10.840380 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:10.843604 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:10.844321 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:10.844327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:10.844334 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:10.844513 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:10.844849 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:10.844852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:10.844854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:10.845352 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:10.845605 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:10.845931 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 20:13:10.846296 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
I20260620 20:13:10.848206 20051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:10.935541 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 19928
I20260620 20:13:10.952813 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 20059
I20260620 20:13:10.971323 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:10.953878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.4.169.131:38967)
0620 20:13:10.954505 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:10.954519 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:10.954562 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:13:10.956187 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:10.956198 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:10.956630 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:10.956924 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:10.958238 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:10.959558 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:10.960769 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:10.961428 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:10.961580 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:10.964615 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:10.964623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:10.964640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:10.964679 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:10.967758 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:10.968583 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:10.968589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:10.968596 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:10.968769 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:10.969174 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:10.969177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:10.969179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:10.969703 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:10.969952 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:10.970555 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 20:13:10.971017 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":248,"threads_started":1}
I20260620 20:13:10.972803 20182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:11.062156 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 20059
2026-06-20T20:13:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24501 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:11 dist-test-slave-4bwq krb5kdc[20245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:11 dist-test-slave-4bwq krb5kdc[20245](info): set up 2 sockets
Jun 20 20:13:11 dist-test-slave-4bwq krb5kdc[20245](info): commencing operation
krb5kdc: starting...
W20260620 20:13:12.179181 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:13:12 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:13:12.413985 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41517
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:12.751582 20258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:12.751943 20258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:12.752166 20258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:12.759599 20258 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:13:12.759734 20258 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:13:12.759835 20258 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:13:12.759933 20258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:12.760035 20258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:12.760159 20258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:12.760248 20258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:12.760344 20258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:12.770617 20258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:12.772004 20258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:12.773351 20258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:12.785403 20264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:12.785472 20263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:12.785454 20266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:12.786166 20258 server_base.cc:1034] running on GCE node
I20260620 20:13:12.787014 20258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:12.788384 20258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:12.789635 20258 hybrid_clock.cc:648] HybridClock initialized: now 1781986392789563 us; error 49 us; skew 500 ppm
I20260620 20:13:12.789978 20258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:12 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986392, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:12.793076 20258 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:13:12.794585 20258 webserver.cc:460] Webserver started at http://127.4.169.190:36189/ using document root <none> and password file <none>
I20260620 20:13:12.795261 20258 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:12.795403 20258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:12.795755 20258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:12.798899 20258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "49ef4c284b08475b8b12ee54a8d16b56"
format_stamp: "Formatted at 2026-06-20 20:13:12 on dist-test-slave-4bwq"
server_key: "caddc1fc2153f1e0ad506731da36194f"
server_key_iv: "6b83b89afbad86909657c63f906b41e5"
server_key_version: "encryptionkey@0"
I20260620 20:13:12.799706 20258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "49ef4c284b08475b8b12ee54a8d16b56"
format_stamp: "Formatted at 2026-06-20 20:13:12 on dist-test-slave-4bwq"
server_key: "caddc1fc2153f1e0ad506731da36194f"
server_key_iv: "6b83b89afbad86909657c63f906b41e5"
server_key_version: "encryptionkey@0"
I20260620 20:13:12.805081 20258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 20:13:12.808928 20273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:12.810264 20258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:13:12.810482 20258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "49ef4c284b08475b8b12ee54a8d16b56"
format_stamp: "Formatted at 2026-06-20 20:13:12 on dist-test-slave-4bwq"
server_key: "caddc1fc2153f1e0ad506731da36194f"
server_key_iv: "6b83b89afbad86909657c63f906b41e5"
server_key_version: "encryptionkey@0"
I20260620 20:13:12.810716 20258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:12.825624 20258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:12.829738 20258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:12.830114 20258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:12.849027 20258 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:41517
I20260620 20:13:12.849051 20324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:41517 every 8 connection(s)
I20260620 20:13:12.850404 20258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:12.854005 20325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:12.857345 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 20258
I20260620 20:13:12.857589 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:13:12.858258 4774 external_mini_cluster.cc:1428] Setting key e0f7ebd60b79dbca877a4d1bf01c3365
I20260620 20:13:12.866060 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Bootstrap starting.
I20260620 20:13:12.869714 20325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:12.870820 20325 log.cc:824] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:12.873624 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: No bootstrap required, opened a new log
Jun 20 20:13:12 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:13:12.884048 20325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:12.884501 20325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:12.884608 20325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef4c284b08475b8b12ee54a8d16b56, State: Initialized, Role: FOLLOWER
I20260620 20:13:12.885241 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:12.860535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.4.169.190:41517)
0620 20:13:12.861512 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:12.861532 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:12.861912 (+ 380us) server_negotiation.cc:407] Connection header received
0620 20:13:12.864520 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:12.864547 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:12.865024 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:12.865441 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:12.866985 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:12.869032 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:12.870683 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:12.871711 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:12.871843 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:12.876568 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:12.876578 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:12.876597 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:12.876647 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:12.880685 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:12.881408 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:12.881422 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:12.881430 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:12.881638 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:12.882030 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:12.882035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:12.882039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:12.882784 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:12.883095 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:12.883472 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 20:13:12.883905 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":434,"threads_started":1}
I20260620 20:13:12.885262 20325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:12.885572 20325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:12.885723 20325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:12.885923 20325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:12.888674 20325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:12.889166 20325 leader_election.cc:302] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49ef4c284b08475b8b12ee54a8d16b56; no voters:
I20260620 20:13:12.889744 20325 leader_election.cc:288] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:12.889942 20330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:12.890923 20330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 1 LEADER]: Becoming Leader. State: Replica: 49ef4c284b08475b8b12ee54a8d16b56, State: Running, Role: LEADER
I20260620 20:13:12.891408 20330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:12.892016 20325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:12.894245 20331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49ef4c284b08475b8b12ee54a8d16b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } } }
I20260620 20:13:12.894268 20332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ef4c284b08475b8b12ee54a8d16b56. Latest consensus state: current_term: 1 leader_uuid: "49ef4c284b08475b8b12ee54a8d16b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } } }
I20260620 20:13:12.894734 20331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:12.894761 20332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:12.896816 20336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:12.901755 20336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:12.912755 20336 catalog_manager.cc:1347] Generated new cluster ID: 7d38f7d7215b49bd8f65b47bfd49723d
I20260620 20:13:12.912914 20336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:12.937254 20336 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:12.938649 20336 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:12.949925 20336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Generated new TSK 0
I20260620 20:13:12.951037 20336 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:13:13.030933 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41517
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:13.373796 20351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:13.374096 20351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:13.374274 20351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:13.381628 20351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:13.381734 20351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:13.381930 20351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:13:13.392009 20351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:13.393533 20351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:13.394963 20351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:13.409001 20356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:13.409040 20359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:13.409087 20357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:13.409818 20351 server_base.cc:1034] running on GCE node
I20260620 20:13:13.410512 20351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:13.411619 20351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:13.412868 20351 hybrid_clock.cc:648] HybridClock initialized: now 1781986393412767 us; error 58 us; skew 500 ppm
I20260620 20:13:13.413247 20351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:13 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:13.416812 20351 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:13:13.418447 20351 webserver.cc:460] Webserver started at http://127.4.169.129:43657/ using document root <none> and password file <none>
I20260620 20:13:13.419198 20351 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:13.419346 20351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:13.419679 20351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:13.422940 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "0fead451fdd24d8987a458ff9d269a3c"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "98f261f0c8311c408bc5afe0009b7191"
server_key_iv: "c2c66c0be5883eb530e01c3e5a171c29"
server_key_version: "encryptionkey@0"
I20260620 20:13:13.423765 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "0fead451fdd24d8987a458ff9d269a3c"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "98f261f0c8311c408bc5afe0009b7191"
server_key_iv: "c2c66c0be5883eb530e01c3e5a171c29"
server_key_version: "encryptionkey@0"
I20260620 20:13:13.429195 20351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 20:13:13.433051 20366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:13.434886 20351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:13:13.435079 20351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "0fead451fdd24d8987a458ff9d269a3c"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "98f261f0c8311c408bc5afe0009b7191"
server_key_iv: "c2c66c0be5883eb530e01c3e5a171c29"
server_key_version: "encryptionkey@0"
I20260620 20:13:13.435288 20351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:13.453742 20351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:13.458243 20351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:13.458596 20351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:13.459635 20351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:13.461086 20351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:13.461192 20351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:13.461349 20351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:13.461423 20351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:13.496447 20351 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:39361
I20260620 20:13:13.496505 20478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:39361 every 8 connection(s)
I20260620 20:13:13.497743 20351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:13.502602 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 20351
I20260620 20:13:13.502841 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:13.503620 4774 external_mini_cluster.cc:1428] Setting key b2d84bdae21b366aa1ef85ca2ab15bbb
Jun 20 20:13:13 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:13:13.521454 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:13.500353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:51165 (local address 127.4.169.190:41517)
0620 20:13:13.500865 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:13.500881 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:13.501820 (+ 939us) server_negotiation.cc:407] Connection header received
0620 20:13:13.504161 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:13.504167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:13.504268 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:13.504434 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:13.506264 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:13.507519 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:13.510045 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:13.510846 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:13.510965 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:13.515690 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:13.515696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:13.515699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:13.515733 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:13.518359 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:13.519136 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:13.519142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:13.519146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:13.519269 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:13.519698 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:13.519704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:13.519708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:13.519993 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:13.520153 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:13.520855 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 20:13:13.521081 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":242,"threads_started":1}
I20260620 20:13:13.522488 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:13.500859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:51165)
0620 20:13:13.501644 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:13.501691 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:13.503874 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:13.504793 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:13.504802 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:13.505323 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:13.506074 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:13.506100 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:13.507700 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:13.507709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:13.509883 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:13.509902 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:13.511111 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:13.511120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:13.511449 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:13.512204 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:13.512239 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:13.515432 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:13.518553 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:13.518563 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:13.518580 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:13.519008 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:13.519387 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:13.519393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:13.519396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:13.519584 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:13.520125 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:13.520133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:13.520482 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 20:13:13.520759 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 20:13:13.521312 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":301,"threads_started":1}
I20260620 20:13:13.524137 20479 heartbeater.cc:347] Connected to a master server at 127.4.169.190:41517
I20260620 20:13:13.524442 20479 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:13.525148 20479 heartbeater.cc:511] Master 127.4.169.190:41517 requested a full tablet report, sending...
I20260620 20:13:13.527371 20290 ts_manager.cc:194] Registered new tserver with Master: 0fead451fdd24d8987a458ff9d269a3c (127.4.169.129:39361)
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:13:13.528913 20290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:51165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:13:13.559717 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41517
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:13.891800 20485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:13.892061 20485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:13.892155 20485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:13.899288 20485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:13.899395 20485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:13.899608 20485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:13:13.909453 20485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:13.910705 20485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:13.912045 20485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:13.924849 20490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:13.924901 20491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:13.924935 20493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:13.925693 20485 server_base.cc:1034] running on GCE node
I20260620 20:13:13.926342 20485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:13.927305 20485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:13.928516 20485 hybrid_clock.cc:648] HybridClock initialized: now 1781986393928479 us; error 32 us; skew 500 ppm
I20260620 20:13:13.928830 20485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:13 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:13.932052 20485 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:13:13.933580 20485 webserver.cc:460] Webserver started at http://127.4.169.130:39545/ using document root <none> and password file <none>
I20260620 20:13:13.934268 20485 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:13.934434 20485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:13.934736 20485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:13.937529 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "2f1c05ef17754f6b491e29e0ba3f3b89"
server_key_iv: "86eb1873d6b2d0c348ab6727721dcc76"
server_key_version: "encryptionkey@0"
I20260620 20:13:13.938216 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "2f1c05ef17754f6b491e29e0ba3f3b89"
server_key_iv: "86eb1873d6b2d0c348ab6727721dcc76"
server_key_version: "encryptionkey@0"
I20260620 20:13:13.943502 20485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:13:13.946946 20500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:13.948274 20485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:13.948451 20485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "2f1c05ef17754f6b491e29e0ba3f3b89"
server_key_iv: "86eb1873d6b2d0c348ab6727721dcc76"
server_key_version: "encryptionkey@0"
I20260620 20:13:13.948670 20485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:13.985016 20485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:13.989645 20485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:13.989959 20485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:13.991032 20485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:13.992331 20485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:13.992414 20485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:13.992583 20485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:13.992652 20485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:14.026504 20485 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:45559
I20260620 20:13:14.026541 20612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:45559 every 8 connection(s)
I20260620 20:13:14.027799 20485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:14.033830 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 20485
I20260620 20:13:14.034039 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:14.034523 4774 external_mini_cluster.cc:1428] Setting key 05362fc53d5f6541633403ca901511a3
Jun 20 20:13:14 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:13:14.050369 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.030308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:39171 (local address 127.4.169.190:41517)
0620 20:13:14.030786 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.030790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.031663 (+ 873us) server_negotiation.cc:407] Connection header received
0620 20:13:14.033480 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.033486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.033577 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.033694 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:14.035396 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.036472 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.039007 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.039467 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.039624 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.043912 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:14.043917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:14.043920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:14.043951 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:14.047022 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.047985 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.047989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.047992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.048123 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.048628 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.048631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.048633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.048918 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:14.049045 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.049933 (+ 888us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.050095 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":276,"threads_started":1}
I20260620 20:13:14.051891 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.030802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:39171)
0620 20:13:14.031512 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:14.031548 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:14.033203 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:14.033920 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:14.033928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:14.034583 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:14.035188 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.035231 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.036646 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.036652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.038827 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.038846 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.039711 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.039720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.040033 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.040753 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:14.040787 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:14.043680 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:14.047163 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:14.047175 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:14.047194 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:14.047823 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:14.048239 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:14.048246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:14.048252 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:14.048498 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:14.049051 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:14.049061 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:14.049518 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 20:13:14.049811 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 20:13:14.050381 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":275,"threads_started":1}
I20260620 20:13:14.053484 20613 heartbeater.cc:347] Connected to a master server at 127.4.169.190:41517
I20260620 20:13:14.053849 20613 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:14.054642 20613 heartbeater.cc:511] Master 127.4.169.190:41517 requested a full tablet report, sending...
I20260620 20:13:14.056186 20290 ts_manager.cc:194] Registered new tserver with Master: 38e0bbe7e0e944aca2b7c1cb4471bf0a (127.4.169.130:45559)
I20260620 20:13:14.057494 20290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:39171
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:13:14.089861 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41517
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:14.413830 20619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:14.414180 20619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:14.414419 20619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:14.421782 20619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:14.421890 20619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:14.422096 20619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:13:14.431898 20619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:14.433158 20619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:14.434531 20619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:14.448001 20625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:14.447947 20624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:14.447990 20627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:14.449010 20619 server_base.cc:1034] running on GCE node
I20260620 20:13:14.449595 20619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:14.450601 20619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:14.451890 20619 hybrid_clock.cc:648] HybridClock initialized: now 1781986394451798 us; error 71 us; skew 500 ppm
I20260620 20:13:14.452162 20619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:14 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986394, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:14.455338 20619 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:13:14.456660 20619 webserver.cc:460] Webserver started at http://127.4.169.131:33117/ using document root <none> and password file <none>
I20260620 20:13:14.457294 20619 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:14.457391 20619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:14.457636 20619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:14.460522 20619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "913784e4458e4deb941873917f9ff593"
format_stamp: "Formatted at 2026-06-20 20:13:14 on dist-test-slave-4bwq"
server_key: "9f534b9e49171e32c0ef947dbe98882e"
server_key_iv: "a897bc9c5974968be2888d8724e6c19b"
server_key_version: "encryptionkey@0"
I20260620 20:13:14.461237 20619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "913784e4458e4deb941873917f9ff593"
format_stamp: "Formatted at 2026-06-20 20:13:14 on dist-test-slave-4bwq"
server_key: "9f534b9e49171e32c0ef947dbe98882e"
server_key_iv: "a897bc9c5974968be2888d8724e6c19b"
server_key_version: "encryptionkey@0"
I20260620 20:13:14.466318 20619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:13:14.469798 20634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:14.471460 20619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:14.471652 20619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "913784e4458e4deb941873917f9ff593"
format_stamp: "Formatted at 2026-06-20 20:13:14 on dist-test-slave-4bwq"
server_key: "9f534b9e49171e32c0ef947dbe98882e"
server_key_iv: "a897bc9c5974968be2888d8724e6c19b"
server_key_version: "encryptionkey@0"
I20260620 20:13:14.471871 20619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:14.497819 20619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:14.502236 20619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:14.502607 20619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:14.503584 20619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:14.504889 20619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:14.504990 20619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:14.505138 20619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:14.505211 20619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:14.531993 20479 heartbeater.cc:503] Master 127.4.169.190:41517 was elected leader, sending a full tablet report...
I20260620 20:13:14.539680 20619 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:45885
I20260620 20:13:14.539716 20746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:45885 every 8 connection(s)
I20260620 20:13:14.540956 20619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:14.543136 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 20619
I20260620 20:13:14.543318 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:14.543880 4774 external_mini_cluster.cc:1428] Setting key b57961b4633d3418eac5be5794b2a204
Jun 20 20:13:14 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986394, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:13:14.560370 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.543414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:55811 (local address 127.4.169.190:41517)
0620 20:13:14.543609 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.543617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.544796 (+ 1179us) server_negotiation.cc:407] Connection header received
0620 20:13:14.546654 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.546659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.546747 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.546866 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:14.548260 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.549004 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.550558 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.550947 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.551054 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.554500 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:14.554505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:14.554508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:14.554551 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:14.557252 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.558143 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.558148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.558151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.558323 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.558824 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.558828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.558829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.559221 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:14.559359 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.559959 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.560126 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:13:14.561259 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.543821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:55811)
0620 20:13:14.544652 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:14.544707 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:14.546353 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:14.547067 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:14.547074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:14.547557 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:14.548113 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.548128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.549121 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.549125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.550364 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.550381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.551130 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.551134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.551320 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.551975 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:14.551996 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:14.554275 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:14.557405 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:14.557411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:14.557422 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:14.558027 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:14.558475 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:14.558478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:14.558480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:14.558711 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:14.559360 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:14.559365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:14.559648 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 20:13:14.559873 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 20:13:14.560223 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":379,"threads_started":1}
I20260620 20:13:14.562505 20747 heartbeater.cc:347] Connected to a master server at 127.4.169.190:41517
I20260620 20:13:14.562803 20747 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:14.563489 20747 heartbeater.cc:511] Master 127.4.169.190:41517 requested a full tablet report, sending...
I20260620 20:13:14.564800 20290 ts_manager.cc:194] Registered new tserver with Master: 913784e4458e4deb941873917f9ff593 (127.4.169.131:45885)
I20260620 20:13:14.565536 20290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:55811
I20260620 20:13:14.571102 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:13:14.588374 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.575538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.4.169.190:41517)
0620 20:13:14.575731 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.575740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.576220 (+ 480us) server_negotiation.cc:407] Connection header received
0620 20:13:14.576402 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.576407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.576474 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.576602 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:14.578060 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.578884 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.580217 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.580621 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.580764 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.583230 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:14.583235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:14.583238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:14.583269 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:14.585692 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.586400 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.586408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.586412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.586554 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.586942 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.586949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.586956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.587245 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:14.587580 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.587827 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.588034 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:13:14.592720 20290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:13:14.600198 20290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:14.627580 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.612565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:45559 (local address 127.0.0.1:49664)
0620 20:13:14.614593 (+ 2028us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:14.614617 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:14.614745 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:14.615880 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:14.615888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:14.615923 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:14.616457 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.616471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.619002 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.619011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.622056 (+ 3045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.622077 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.626288 (+ 4211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.626297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.626627 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.626700 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 20:13:14.626881 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 20:13:14.627155 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1880,"thread_start_us":1711,"threads_started":1}
I20260620 20:13:14.628320 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.612426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.4.169.131:45885)
0620 20:13:14.613509 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.613520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.614031 (+ 511us) server_negotiation.cc:407] Connection header received
0620 20:13:14.614476 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.614484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.614755 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.614936 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:14.615999 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.618044 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.624249 (+ 6205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.626216 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.626402 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.626698 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.627699 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.627867 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":724,"threads_started":1}
I20260620 20:13:14.629331 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.612143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:45885 (local address 127.0.0.1:51912)
0620 20:13:14.613879 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:14.613928 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:14.614189 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:14.615319 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:14.615326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:14.615359 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:14.615785 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.615801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.618281 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.618289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.624008 (+ 5719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.624027 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.626905 (+ 2878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.626914 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.627201 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.627253 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 20:13:14.627374 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 20:13:14.627570 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1482,"mutex_wait_us":2626,"thread_start_us":1323,"threads_started":1}
I20260620 20:13:14.629547 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.613250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.4.169.130:45559)
0620 20:13:14.613978 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.613988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.614884 (+ 896us) server_negotiation.cc:407] Connection header received
0620 20:13:14.614996 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.615007 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.615384 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.615596 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:14.616708 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.618793 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.622270 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.625669 (+ 3399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.625842 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.626107 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.628106 (+ 1999us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.629206 (+ 1100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":382,"threads_started":1}
I20260620 20:13:14.633424 20682 tablet_service.cc:1467] Processing CreateTablet for tablet 1db0918459334b90b37256a5cfc4c53b (DEFAULT_TABLE table=test-table [id=c7ff16d170744b8790cf264a6eca7493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:14.633424 20548 tablet_service.cc:1467] Processing CreateTablet for tablet 1db0918459334b90b37256a5cfc4c53b (DEFAULT_TABLE table=test-table [id=c7ff16d170744b8790cf264a6eca7493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:14.634220 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.614832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39361 (local address 127.0.0.1:34274)
0620 20:13:14.627799 (+ 12967us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:14.627820 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:14.627935 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:14.629003 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:14.629011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:14.629036 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:14.629309 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.629323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.631114 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.631122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.632250 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.632261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.633483 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.633489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.633682 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.633719 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:13:14.633831 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:13:14.633963 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12887,"thread_start_us":13930,"threads_started":1}
I20260620 20:13:14.634680 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.615246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.4.169.129:39361)
0620 20:13:14.627622 (+ 12376us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.627633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.627891 (+ 258us) server_negotiation.cc:407] Connection header received
0620 20:13:14.628085 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.628093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.628323 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.628516 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:14.629460 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.630988 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.632418 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.633321 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.633472 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.633689 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.633878 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.634031 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12190,"thread_start_us":12389,"threads_started":1}
I20260620 20:13:14.634935 20548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db0918459334b90b37256a5cfc4c53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:14.634938 20682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db0918459334b90b37256a5cfc4c53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:14.637745 20414 tablet_service.cc:1467] Processing CreateTablet for tablet 1db0918459334b90b37256a5cfc4c53b (DEFAULT_TABLE table=test-table [id=c7ff16d170744b8790cf264a6eca7493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:14.639060 20414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db0918459334b90b37256a5cfc4c53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:14.649819 20767 tablet_bootstrap.cc:492] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: Bootstrap starting.
I20260620 20:13:14.650889 20766 tablet_bootstrap.cc:492] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: Bootstrap starting.
I20260620 20:13:14.651978 20765 tablet_bootstrap.cc:492] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Bootstrap starting.
I20260620 20:13:14.654803 20766 tablet_bootstrap.cc:654] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:14.655071 20765 tablet_bootstrap.cc:654] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:14.655427 20767 tablet_bootstrap.cc:654] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:14.656104 20765 log.cc:824] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:14.656162 20766 log.cc:824] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:14.656571 20767 log.cc:824] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:14.659078 20765 tablet_bootstrap.cc:492] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: No bootstrap required, opened a new log
I20260620 20:13:14.659364 20765 ts_tablet_manager.cc:1397] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 20:13:14.659466 20767 tablet_bootstrap.cc:492] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: No bootstrap required, opened a new log
I20260620 20:13:14.659720 20767 ts_tablet_manager.cc:1397] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 20:13:14.660271 20766 tablet_bootstrap.cc:492] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: No bootstrap required, opened a new log
I20260620 20:13:14.660521 20766 ts_tablet_manager.cc:1397] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 20:13:14.667342 20765 raft_consensus.cc:348] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:14.667762 20765 raft_consensus.cc:374] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:14.667899 20765 raft_consensus.cc:729] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e0bbe7e0e944aca2b7c1cb4471bf0a, State: Initialized, Role: FOLLOWER
I20260620 20:13:14.668560 20765 consensus_queue.cc:260] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:14.669723 20613 heartbeater.cc:503] Master 127.4.169.190:41517 was elected leader, sending a full tablet report...
I20260620 20:13:14.670253 20765 ts_tablet_manager.cc:1428] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 20:13:14.671418 20766 raft_consensus.cc:348] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:14.671847 20766 raft_consensus.cc:374] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:14.671958 20766 raft_consensus.cc:729] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913784e4458e4deb941873917f9ff593, State: Initialized, Role: FOLLOWER
I20260620 20:13:14.672216 20767 raft_consensus.cc:348] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:14.672637 20767 raft_consensus.cc:374] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:14.672736 20767 raft_consensus.cc:729] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fead451fdd24d8987a458ff9d269a3c, State: Initialized, Role: FOLLOWER
I20260620 20:13:14.672752 20766 consensus_queue.cc:260] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:14.673463 20767 consensus_queue.cc:260] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:14.674034 20747 heartbeater.cc:503] Master 127.4.169.190:41517 was elected leader, sending a full tablet report...
I20260620 20:13:14.674803 20766 ts_tablet_manager.cc:1428] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260620 20:13:14.676560 20767 ts_tablet_manager.cc:1428] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260620 20:13:14.690181 20772 raft_consensus.cc:482] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:14.690474 20772 raft_consensus.cc:504] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:14.692091 20772 leader_election.cc:288] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38e0bbe7e0e944aca2b7c1cb4471bf0a (127.4.169.130:45559), 0fead451fdd24d8987a458ff9d269a3c (127.4.169.129:39361)
I20260620 20:13:14.698794 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.692387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:46147 (local address 127.4.169.130:45559)
0620 20:13:14.692610 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.692616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.692766 (+ 150us) server_negotiation.cc:407] Connection header received
0620 20:13:14.693021 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.693027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.693116 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.693209 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:14.693826 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.694930 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.696797 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.697987 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.698094 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.698144 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.698526 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.698615 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:13:14.698920 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.692447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:45559 (local address 127.4.169.131:46147)
0620 20:13:14.692656 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:14.692677 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:14.692814 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:14.693394 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:14.693402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:14.693453 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:14.693674 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.693683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.695061 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.695067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.696635 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.696647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.698130 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.698138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.698329 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.698366 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:13:14.698475 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 20:13:14.698613 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 20:13:14.699781 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.693111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:43777 (local address 127.4.169.129:39361)
0620 20:13:14.693293 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.693299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.693866 (+ 567us) server_negotiation.cc:407] Connection header received
0620 20:13:14.694021 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.694025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.694078 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.694146 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:14.694745 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.695770 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.697823 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.698943 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.699107 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.699190 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.699509 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.699621 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:13:14.699941 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.692873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39361 (local address 127.4.169.131:43777)
0620 20:13:14.693725 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:14.693772 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:14.693909 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:14.694188 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:14.694192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:14.694217 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:14.694597 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.694608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.695926 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.695930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.697666 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:14.697681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.699175 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.699183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:14.699356 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.699386 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:13:14.699457 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 20:13:14.699569 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":1,"thread_start_us":553,"threads_started":1}
I20260620 20:13:14.700093 20568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db0918459334b90b37256a5cfc4c53b" candidate_uuid: "913784e4458e4deb941873917f9ff593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" is_pre_election: true
I20260620 20:13:14.700587 20568 raft_consensus.cc:2454] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913784e4458e4deb941873917f9ff593 in term 0.
I20260620 20:13:14.701056 20434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db0918459334b90b37256a5cfc4c53b" candidate_uuid: "913784e4458e4deb941873917f9ff593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fead451fdd24d8987a458ff9d269a3c" is_pre_election: true
I20260620 20:13:14.701256 20637 leader_election.cc:302] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e0bbe7e0e944aca2b7c1cb4471bf0a, 913784e4458e4deb941873917f9ff593; no voters:
I20260620 20:13:14.701634 20772 raft_consensus.cc:2781] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:14.701589 20434 raft_consensus.cc:2454] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913784e4458e4deb941873917f9ff593 in term 0.
I20260620 20:13:14.701746 20772 raft_consensus.cc:482] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:14.701879 20772 raft_consensus.cc:3037] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:14.704316 20772 raft_consensus.cc:504] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:14.705009 20772 leader_election.cc:288] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [CANDIDATE]: Term 1 election: Requested vote from peers 38e0bbe7e0e944aca2b7c1cb4471bf0a (127.4.169.130:45559), 0fead451fdd24d8987a458ff9d269a3c (127.4.169.129:39361)
I20260620 20:13:14.705416 20568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db0918459334b90b37256a5cfc4c53b" candidate_uuid: "913784e4458e4deb941873917f9ff593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a"
I20260620 20:13:14.705493 20434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db0918459334b90b37256a5cfc4c53b" candidate_uuid: "913784e4458e4deb941873917f9ff593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fead451fdd24d8987a458ff9d269a3c"
I20260620 20:13:14.705598 20568 raft_consensus.cc:3037] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:14.705739 20434 raft_consensus.cc:3037] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:14.708062 20434 raft_consensus.cc:2454] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913784e4458e4deb941873917f9ff593 in term 1.
I20260620 20:13:14.708062 20568 raft_consensus.cc:2454] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913784e4458e4deb941873917f9ff593 in term 1.
I20260620 20:13:14.708451 20637 leader_election.cc:302] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fead451fdd24d8987a458ff9d269a3c, 913784e4458e4deb941873917f9ff593; no voters:
I20260620 20:13:14.708734 20772 raft_consensus.cc:2781] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:14.709164 20772 raft_consensus.cc:686] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 1 LEADER]: Becoming Leader. State: Replica: 913784e4458e4deb941873917f9ff593, State: Running, Role: LEADER
I20260620 20:13:14.709612 20772 consensus_queue.cc:237] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:14.714121 20290 catalog_manager.cc:5495] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 reported cstate change: term changed from 0 to 1, leader changed from <none> to 913784e4458e4deb941873917f9ff593 (127.4.169.131). New cstate: current_term: 1 leader_uuid: "913784e4458e4deb941873917f9ff593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } health_report { overall_health: UNKNOWN } } }
I20260620 20:13:14.749411 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.743878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.4.169.131:45885)
0620 20:13:14.744080 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.744088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.744182 (+ 94us) server_negotiation.cc:407] Connection header received
0620 20:13:14.744438 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.744442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.744533 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.744690 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:14.745073 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.745796 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.747322 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.747726 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.747863 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.748355 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:14.748845 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:14.749008 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.749060 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.749141 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 20:13:14.750103 20480 tablet.cc:2369] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:14.757496 20434 raft_consensus.cc:1261] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [term 1 FOLLOWER]: Refusing update from remote peer 913784e4458e4deb941873917f9ff593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:14.757498 20568 raft_consensus.cc:1261] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 1 FOLLOWER]: Refusing update from remote peer 913784e4458e4deb941873917f9ff593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:14.758440 20775 consensus_queue.cc:1035] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:14.758795 20772 consensus_queue.cc:1035] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:14.783681 20290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:14.783948 20290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:14.787328 20290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Sending DeleteTablet for 3 replicas of tablet 1db0918459334b90b37256a5cfc4c53b
I20260620 20:13:14.788419 20682 tablet_service.cc:1514] Processing DeleteTablet for tablet 1db0918459334b90b37256a5cfc4c53b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:14 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:51912
I20260620 20:13:14.788592 20548 tablet_service.cc:1514] Processing DeleteTablet for tablet 1db0918459334b90b37256a5cfc4c53b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:14 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:49664
I20260620 20:13:14.788895 20414 tablet_service.cc:1514] Processing DeleteTablet for tablet 1db0918459334b90b37256a5cfc4c53b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:14 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:34274
I20260620 20:13:14.789430 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 20351
I20260620 20:13:14.799042 20790 tablet_replica.cc:330] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: stopping tablet replica
I20260620 20:13:14.799732 20790 raft_consensus.cc:2229] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:14.800426 20790 raft_consensus.cc:2258] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:14.801578 20788 tablet_replica.cc:330] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: stopping tablet replica
I20260620 20:13:14.802027 20788 raft_consensus.cc:2229] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:14.802906 20788 raft_consensus.cc:2258] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:14.803304 20790 ts_tablet_manager.cc:1905] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:14.805403 20788 ts_tablet_manager.cc:1905] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:14.810547 20790 ts_tablet_manager.cc:1918] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:14.810755 20790 log.cc:1196] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/1db0918459334b90b37256a5cfc4c53b
I20260620 20:13:14.811378 20790 ts_tablet_manager.cc:1939] T 1db0918459334b90b37256a5cfc4c53b P 0fead451fdd24d8987a458ff9d269a3c: Deleting consensus metadata
I20260620 20:13:14.813333 20276 catalog_manager.cc:4841] TS 0fead451fdd24d8987a458ff9d269a3c (127.4.169.129:39361): tablet 1db0918459334b90b37256a5cfc4c53b (table test-table [id=c7ff16d170744b8790cf264a6eca7493]) successfully deleted
I20260620 20:13:14.815135 20789 tablet_replica.cc:330] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: stopping tablet replica
I20260620 20:13:14.815634 20789 raft_consensus.cc:2229] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:14.816205 20789 raft_consensus.cc:2258] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:14.816220 20788 ts_tablet_manager.cc:1918] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:14.816507 20788 log.cc:1196] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/1db0918459334b90b37256a5cfc4c53b
I20260620 20:13:14.817394 20788 ts_tablet_manager.cc:1939] T 1db0918459334b90b37256a5cfc4c53b P 913784e4458e4deb941873917f9ff593: Deleting consensus metadata
I20260620 20:13:14.818322 20789 ts_tablet_manager.cc:1905] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:14.819268 20277 catalog_manager.cc:4841] TS 913784e4458e4deb941873917f9ff593 (127.4.169.131:45885): tablet 1db0918459334b90b37256a5cfc4c53b (table test-table [id=c7ff16d170744b8790cf264a6eca7493]) successfully deleted
Jun 20 20:13:14 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:13:14.825413 20789 ts_tablet_manager.cc:1918] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:14.825598 20789 log.cc:1196] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/1db0918459334b90b37256a5cfc4c53b
I20260620 20:13:14.826104 20789 ts_tablet_manager.cc:1939] T 1db0918459334b90b37256a5cfc4c53b P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Deleting consensus metadata
I20260620 20:13:14.828001 20276 catalog_manager.cc:4841] TS 38e0bbe7e0e944aca2b7c1cb4471bf0a (127.4.169.130:45559): tablet 1db0918459334b90b37256a5cfc4c53b (table test-table [id=c7ff16d170744b8790cf264a6eca7493]) successfully deleted
I20260620 20:13:14.831341 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.790813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.4.169.129:39361)
0620 20:13:14.791051 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.791060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.814279 (+ 23219us) server_negotiation.cc:407] Connection header received
0620 20:13:14.814518 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.814524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.814603 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.814726 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:14.816988 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.818108 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.820016 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.820475 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.820628 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.824945 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:14.824961 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:14.824975 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:14.825009 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:14.828320 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.829114 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.829129 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.829131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.829265 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.829674 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.829679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.829684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.830034 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:14.830348 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.830752 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.830984 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:13:14.831993 20474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:14.922587 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 20351
I20260620 20:13:14.946157 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 20485
Jun 20 20:13:14 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:13:14.963670 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:14.947127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.4.169.130:45559)
0620 20:13:14.947335 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:14.947342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:14.947372 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:13:14.947440 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:14.947445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:14.947534 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:14.947648 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:14.948999 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.950154 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.951682 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:14.952311 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:14.952447 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:14.956645 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:14.956663 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:14.956683 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:14.956725 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:14.960605 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.961330 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.961339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.961343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.961509 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:14.961867 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:14.961872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:14.961876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:14.962321 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:14.962668 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:14.963066 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 20:13:14.963288 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:13:14.964418 20608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:15.057605 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 20485
I20260620 20:13:15.074514 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 20619
Jun 20 20:13:15 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:13:15.094321 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:15.075478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.4.169.131:45885)
0620 20:13:15.075686 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:15.075692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:15.075811 (+ 119us) server_negotiation.cc:407] Connection header received
0620 20:13:15.076032 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:15.076035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:15.076125 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:15.076218 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:15.078146 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:15.079068 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:15.081171 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:15.081623 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:15.081719 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:15.087525 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:15.087537 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:15.087551 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:15.087602 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:15.091099 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:15.091955 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:15.091962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:15.091965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:15.092139 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:15.092626 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:15.092631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:15.092636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:15.093004 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:15.093342 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:15.093697 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 20:13:15.093932 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:13:15.095191 20742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:15.219797 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 20619
I20260620 20:13:15.239059 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 20258
I20260620 20:13:15.240175 20320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:15.350220 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 20258
I20260620 20:13:35.371696 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41517
--webserver_interface=127.4.169.190
--webserver_port=36189
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:35.732646 20796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:35.733021 20796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:35.733199 20796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:35.740727 20796 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:13:35.740888 20796 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:13:35.740994 20796 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:13:35.741094 20796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:35.741199 20796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:35.741322 20796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:35.741413 20796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:35.741494 20796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:35.751893 20796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=36189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:35.753304 20796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:35.754737 20796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:35.779712 20803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:35.779714 20805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:35.779738 20802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:35.780021 20796 server_base.cc:1034] running on GCE node
I20260620 20:13:35.781298 20796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:35.783172 20796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:35.784421 20796 hybrid_clock.cc:648] HybridClock initialized: now 1781986415784351 us; error 38 us; skew 500 ppm
I20260620 20:13:35.784891 20796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:35 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986415, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:35.788868 20796 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:13:35.790645 20796 webserver.cc:460] Webserver started at http://127.4.169.190:36189/ using document root <none> and password file <none>
I20260620 20:13:35.791497 20796 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:35.791643 20796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:35.798060 20796 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 20:13:35.801451 20812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:35.803151 20796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:35.803361 20796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "49ef4c284b08475b8b12ee54a8d16b56"
format_stamp: "Formatted at 2026-06-20 20:13:12 on dist-test-slave-4bwq"
server_key: "caddc1fc2153f1e0ad506731da36194f"
server_key_iv: "6b83b89afbad86909657c63f906b41e5"
server_key_version: "encryptionkey@0"
I20260620 20:13:35.804546 20796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:35.823602 20796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:35.828229 20796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:35.828543 20796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:35.848644 20796 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:41517
I20260620 20:13:35.848668 20863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:41517 every 8 connection(s)
I20260620 20:13:35.850278 20796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:35.857460 20864 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:13:35.857869 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 20796
I20260620 20:13:35.858814 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:39361
--local_ip_for_outbound_sockets=127.4.169.129
--tserver_master_addrs=127.4.169.190:41517
--webserver_port=43657
--webserver_interface=127.4.169.129
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:13:35.861354 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Bootstrap starting.
I20260620 20:13:35.882580 20864 log.cc:824] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:35.894563 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:13:35.895090 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Bootstrap complete.
I20260620 20:13:35.904345 20864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:35.905193 20864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef4c284b08475b8b12ee54a8d16b56, State: Initialized, Role: FOLLOWER
I20260620 20:13:35.905819 20864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:35.906092 20864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:35.906224 20864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:35.906436 20864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:13:35.910528 20864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:35.911041 20864 leader_election.cc:302] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49ef4c284b08475b8b12ee54a8d16b56; no voters:
I20260620 20:13:35.911621 20864 leader_election.cc:288] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:13:35.911828 20868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:13:35.913033 20868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 2 LEADER]: Becoming Leader. State: Replica: 49ef4c284b08475b8b12ee54a8d16b56, State: Running, Role: LEADER
I20260620 20:13:35.913600 20868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:35.913798 20864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:35.916221 20870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ef4c284b08475b8b12ee54a8d16b56. Latest consensus state: current_term: 2 leader_uuid: "49ef4c284b08475b8b12ee54a8d16b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } } }
I20260620 20:13:35.916676 20870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:35.917297 20869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "49ef4c284b08475b8b12ee54a8d16b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } } }
I20260620 20:13:35.917661 20869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:35.927915 20875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:35.938697 20875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:35.940735 20875 catalog_manager.cc:1259] Loaded cluster ID: 7d38f7d7215b49bd8f65b47bfd49723d
I20260620 20:13:35.940953 20875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:35.946261 20875 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:35.948546 20875 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Loaded TSK: 0
I20260620 20:13:35.950197 20885 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:13:35.971364 20875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Generated new TSK 1
I20260620 20:13:35.971794 20875 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:13:36.236665 20866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:36.236986 20866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:36.237142 20866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:36.244506 20866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:36.244671 20866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:36.244843 20866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:13:36.254961 20866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:39361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=43657
--tserver_master_addrs=127.4.169.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:36.256312 20866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:36.257706 20866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:36.272744 20891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:36.272765 20892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:36.273756 20894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:36.273815 20866 server_base.cc:1034] running on GCE node
I20260620 20:13:36.274585 20866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:36.275609 20866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:36.276839 20866 hybrid_clock.cc:648] HybridClock initialized: now 1781986416276778 us; error 58 us; skew 500 ppm
I20260620 20:13:36.277187 20866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:36 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:36.280638 20866 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:13:36.282379 20866 webserver.cc:460] Webserver started at http://127.4.169.129:43657/ using document root <none> and password file <none>
I20260620 20:13:36.283169 20866 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:36.283339 20866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:36.288988 20866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:13:36.292026 20901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:36.293565 20866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:36.293771 20866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "0fead451fdd24d8987a458ff9d269a3c"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "98f261f0c8311c408bc5afe0009b7191"
server_key_iv: "c2c66c0be5883eb530e01c3e5a171c29"
server_key_version: "encryptionkey@0"
I20260620 20:13:36.294875 20866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:36.327309 20866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:36.332039 20866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:36.332342 20866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:36.333289 20866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:36.334719 20866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:36.334848 20866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:36.334947 20866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:36.335026 20866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:36.368700 20866 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:39361
I20260620 20:13:36.368788 21013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:39361 every 8 connection(s)
I20260620 20:13:36.370213 20866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:36.374717 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 20866
I20260620 20:13:36.375375 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:45559
--local_ip_for_outbound_sockets=127.4.169.130
--tserver_master_addrs=127.4.169.190:41517
--webserver_port=39545
--webserver_interface=127.4.169.130
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:13:36 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:13:36.395992 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:36.373448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:57481)
0620 20:13:36.374373 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:36.374465 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:36.376364 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:36.378037 (+ 1673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:36.378045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:36.378639 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:36.379207 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:36.379224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.381665 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.381669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:36.383342 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:36.383374 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.384460 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.384468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:36.384744 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:36.385459 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:36.385482 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:36.388002 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:36.391576 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:36.391586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:36.391601 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:36.392034 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:36.392481 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:36.392486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:36.392490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:36.392732 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:36.393498 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:36.393506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:36.393838 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 20:13:36.394103 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 20:13:36.394676 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":454,"threads_started":1}
I20260620 20:13:36.395983 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:36.373362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:57481 (local address 127.4.169.190:41517)
0620 20:13:36.374024 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:36.374035 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:36.374618 (+ 583us) server_negotiation.cc:407] Connection header received
0620 20:13:36.376982 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:36.377004 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:36.377475 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:36.377871 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:36.379422 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.381484 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.383513 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.384190 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.384419 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:36.388297 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:36.388305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:36.388319 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:36.388367 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:36.391436 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:36.392182 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:36.392189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:36.392195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:36.392348 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:36.392865 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:36.392870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:36.392873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:36.393341 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:36.393558 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:36.394320 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 20:13:36.394749 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":299,"threads_started":1}
I20260620 20:13:36.399294 21014 heartbeater.cc:347] Connected to a master server at 127.4.169.190:41517
I20260620 20:13:36.399632 21014 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:36.400426 21014 heartbeater.cc:511] Master 127.4.169.190:41517 requested a full tablet report, sending...
I20260620 20:13:36.402938 20829 ts_manager.cc:194] Registered new tserver with Master: 0fead451fdd24d8987a458ff9d269a3c (127.4.169.129:39361)
I20260620 20:13:36.404983 20829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:57481
W20260620 20:13:36.709024 21018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:36.709300 21018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:36.709393 21018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:36.716709 21018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:36.716830 21018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:36.716957 21018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:13:36.726544 21018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:45559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=39545
--tserver_master_addrs=127.4.169.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:36.727775 21018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:36.729238 21018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:36.742247 21023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:36.742213 21026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:36.742213 21024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:36.743541 21018 server_base.cc:1034] running on GCE node
I20260620 20:13:36.744217 21018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:36.745189 21018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:36.746481 21018 hybrid_clock.cc:648] HybridClock initialized: now 1781986416746452 us; error 63 us; skew 500 ppm
I20260620 20:13:36.746852 21018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:36 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:36.750298 21018 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:13:36.751812 21018 webserver.cc:460] Webserver started at http://127.4.169.130:39545/ using document root <none> and password file <none>
I20260620 20:13:36.752523 21018 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:36.752682 21018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:36.758137 21018 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:13:36.761046 21033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:36.762336 21018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:13:36.762576 21018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "2f1c05ef17754f6b491e29e0ba3f3b89"
server_key_iv: "86eb1873d6b2d0c348ab6727721dcc76"
server_key_version: "encryptionkey@0"
I20260620 20:13:36.763577 21018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:36.776445 21018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:36.780676 21018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:36.781036 21018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:36.782362 21018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:36.783758 21018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:36.783908 21018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:36.784040 21018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:36.784142 21018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:36.818859 21018 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:45559
I20260620 20:13:36.818898 21145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:45559 every 8 connection(s)
I20260620 20:13:36.820348 21018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:36.829708 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 21018
I20260620 20:13:36.830542 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:45885
--local_ip_for_outbound_sockets=127.4.169.131
--tserver_master_addrs=127.4.169.190:41517
--webserver_port=33117
--webserver_interface=127.4.169.131
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:13:36 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:13:36.841673 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:36.822911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:51215 (local address 127.4.169.190:41517)
0620 20:13:36.823100 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:36.823104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:36.824211 (+ 1107us) server_negotiation.cc:407] Connection header received
0620 20:13:36.826012 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:36.826017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:36.826088 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:36.826179 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:36.827787 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.828552 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.830931 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.831380 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.831508 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:36.835845 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:36.835851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:36.835854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:36.835891 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:36.838509 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:36.839264 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:36.839272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:36.839280 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:36.839408 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:36.839890 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:36.839905 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:36.839907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:36.840170 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:36.840340 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:36.841063 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 20:13:36.841321 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:13:36.842883 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:36.823344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:51215)
0620 20:13:36.824050 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:36.824092 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:36.825747 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:36.826454 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:36.826462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:36.827015 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:36.827609 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:36.827627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.828690 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.828694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:36.830710 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:36.830737 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.831652 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.831662 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:36.832025 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:36.832792 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:36.832828 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:36.835594 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:36.838652 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:36.838662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:36.838679 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:36.839134 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:36.839553 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:36.839559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:36.839563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:36.839768 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:36.840301 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:36.840309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:36.840668 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 20:13:36.840975 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 20:13:36.841541 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":269,"threads_started":1}
I20260620 20:13:36.844465 21146 heartbeater.cc:347] Connected to a master server at 127.4.169.190:41517
I20260620 20:13:36.844774 21146 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:36.845505 21146 heartbeater.cc:511] Master 127.4.169.190:41517 requested a full tablet report, sending...
I20260620 20:13:36.847025 20829 ts_manager.cc:194] Registered new tserver with Master: 38e0bbe7e0e944aca2b7c1cb4471bf0a (127.4.169.130:45559)
I20260620 20:13:36.847946 20829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:51215
W20260620 20:13:37.171837 21149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:37.172114 21149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:37.172290 21149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:37.179806 21149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:37.179935 21149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:37.180121 21149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:13:37.190331 21149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:45885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=33117
--tserver_master_addrs=127.4.169.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:37.191682 21149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:37.193065 21149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:37.206534 21154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:37.206518 21155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:37.206619 21157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:37.207239 21149 server_base.cc:1034] running on GCE node
I20260620 20:13:37.207882 21149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:37.208878 21149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:37.210093 21149 hybrid_clock.cc:648] HybridClock initialized: now 1781986417210048 us; error 40 us; skew 500 ppm
I20260620 20:13:37.210351 21149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:37 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:37.213512 21149 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:13:37.215077 21149 webserver.cc:460] Webserver started at http://127.4.169.131:33117/ using document root <none> and password file <none>
I20260620 20:13:37.215811 21149 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:37.215957 21149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:37.221127 21149 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:13:37.223775 21164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:37.225039 21149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:13:37.225216 21149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "913784e4458e4deb941873917f9ff593"
format_stamp: "Formatted at 2026-06-20 20:13:14 on dist-test-slave-4bwq"
server_key: "9f534b9e49171e32c0ef947dbe98882e"
server_key_iv: "a897bc9c5974968be2888d8724e6c19b"
server_key_version: "encryptionkey@0"
I20260620 20:13:37.226169 21149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:37.246438 21149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:37.250355 21149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:37.250671 21149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:37.251648 21149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:37.253002 21149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:37.253124 21149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:37.253269 21149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:37.253357 21149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:37.287072 21149 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:45885
I20260620 20:13:37.287125 21276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:45885 every 8 connection(s)
I20260620 20:13:37.288471 21149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:37.295832 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 21149
Jun 20 20:13:37 dist-test-slave-4bwq krb5kdc[20245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:13:37.310322 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.290927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:34183 (local address 127.4.169.190:41517)
0620 20:13:37.291099 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.291103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.292121 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 20:13:37.293895 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.293901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.294000 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.294118 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:37.295600 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.296418 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.298480 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.299230 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.299367 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.303164 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:37.303169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:37.303172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:37.303203 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:37.306745 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.307668 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.307672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.307674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.307791 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.308307 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.308312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.308316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.308714 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:37.308871 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.309678 (+ 807us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.309905 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:13:37.310995 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.291309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:34183)
0620 20:13:37.291971 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:37.292006 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:37.293637 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:37.294351 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:37.294359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:37.294893 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:37.295427 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.295454 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.296608 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.296614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.298312 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.298324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.299433 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.299438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.299702 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.300312 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:37.300347 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:37.302946 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:37.306890 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:37.306896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:37.306907 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:37.307530 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:37.307935 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:37.307940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:37.307944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:37.308130 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:37.308897 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:37.308905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:37.309262 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 20:13:37.309545 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 20:13:37.309904 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":257,"threads_started":1}
I20260620 20:13:37.312515 21277 heartbeater.cc:347] Connected to a master server at 127.4.169.190:41517
I20260620 20:13:37.312812 21277 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:37.313530 21277 heartbeater.cc:511] Master 127.4.169.190:41517 requested a full tablet report, sending...
I20260620 20:13:37.313997 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.296753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.4.169.190:41517)
0620 20:13:37.297257 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.297263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.297574 (+ 311us) server_negotiation.cc:407] Connection header received
0620 20:13:37.297700 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.297707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.297777 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.297885 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:37.299995 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.301108 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.302993 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.303524 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.303620 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.307064 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:37.307072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:37.307077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:37.307123 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:37.310355 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.312010 (+ 1655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.312015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.312018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.312151 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.312543 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.312546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.312548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.312834 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:37.313107 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.313417 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.313637 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":260,"threads_started":1}
I20260620 20:13:37.315261 20828 ts_manager.cc:194] Registered new tserver with Master: 913784e4458e4deb941873917f9ff593 (127.4.169.131:45885)
I20260620 20:13:37.316288 20828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:34183
I20260620 20:13:37.316766 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:13:37.322999 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.317680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.4.169.190:41517)
0620 20:13:37.317862 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.317868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.318337 (+ 469us) server_negotiation.cc:407] Connection header received
0620 20:13:37.318562 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.318568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.318735 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.318888 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:37.319221 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.319880 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.321345 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.321727 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.321814 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.322301 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:37.322547 (+ 246us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 20:13:37.322808 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45018: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 20:13:37.323127 21280 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45018: token expired
W20260620 20:13:37.323727 20750 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.169.190:41517)
I20260620 20:13:37.336309 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.324187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.4.169.190:41517)
0620 20:13:37.324328 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.324332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.324345 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:13:37.324380 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.324383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.324429 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.324504 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:37.325759 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.326495 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.327765 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.328125 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.328271 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.331253 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:37.331263 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:37.331266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:37.331296 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:37.333650 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.334315 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.334320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.334324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.334522 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.334890 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.334896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.334900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.335214 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:37.335474 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.335762 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.335977 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:13:37.339932 20828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:13:37.347118 20828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:37.372560 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.358533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.4.169.130:45559)
0620 20:13:37.360690 (+ 2157us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.360699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.360728 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:13:37.360849 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.360860 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.361175 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.362328 (+ 1153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:37.362627 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.364728 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.369094 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.370786 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.371703 (+ 917us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.371888 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.371958 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.372190 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2007,"thread_start_us":1868,"threads_started":1}
I20260620 20:13:37.372885 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.359220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.4.169.131:45885)
0620 20:13:37.360232 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.360241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.360624 (+ 383us) server_negotiation.cc:407] Connection header received
0620 20:13:37.360849 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.360860 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.361157 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.361368 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:37.362634 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.365021 (+ 2387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.369067 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.370788 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.370924 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.371175 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.372364 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.372525 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":547,"threads_started":1}
I20260620 20:13:37.372150 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.358493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:45885 (local address 127.0.0.1:41292)
0620 20:13:37.360512 (+ 2019us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:37.360528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:37.360652 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:37.361725 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:37.361733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:37.361773 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:37.362353 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.362367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.365234 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.365242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.368918 (+ 3676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.368933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.371321 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.371326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.371503 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.371532 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:13:37.371597 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 20:13:37.371733 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1885,"mutex_wait_us":778,"thread_start_us":385,"threads_started":1}
I20260620 20:13:37.373692 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.358436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:45559 (local address 127.0.0.1:51798)
0620 20:13:37.359215 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:37.359256 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:37.359414 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:37.361464 (+ 2050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:37.361475 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:37.361512 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:37.362088 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.362105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.364935 (+ 2830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.364944 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.368310 (+ 3366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.368328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.371048 (+ 2720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.371055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.371270 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.371342 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 20:13:37.371462 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 20:13:37.373196 (+ 1734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":1517,"thread_start_us":1746,"threads_started":1}
I20260620 20:13:37.377277 21081 tablet_service.cc:1467] Processing CreateTablet for tablet 017f28afa7294822a70d03d6cceb79c2 (DEFAULT_TABLE table=test-table [id=32d0f9a9ef5c4d23a8cb1bdb23a96edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:37.377283 21212 tablet_service.cc:1467] Processing CreateTablet for tablet 017f28afa7294822a70d03d6cceb79c2 (DEFAULT_TABLE table=test-table [id=32d0f9a9ef5c4d23a8cb1bdb23a96edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:37.378526 21081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 017f28afa7294822a70d03d6cceb79c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:37.378657 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 017f28afa7294822a70d03d6cceb79c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:37.378839 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.360631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39361 (local address 127.0.0.1:37006)
0620 20:13:37.368371 (+ 7740us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:37.368390 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:37.368530 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:37.370605 (+ 2075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:37.370609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:37.370639 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:37.371051 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.371062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.374156 (+ 3094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.374161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.376194 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.376208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.378085 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.378092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.378279 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.378319 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 20:13:37.378410 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 20:13:37.378534 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7532,"thread_start_us":8436,"threads_started":1}
I20260620 20:13:37.379741 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.361047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.4.169.129:39361)
0620 20:13:37.369996 (+ 8949us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.370005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.370033 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:13:37.370119 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.370126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.370366 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.371242 (+ 876us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:37.371474 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.374005 (+ 2531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.376421 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.377828 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.378787 (+ 959us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.379165 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.379233 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.379384 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7923,"thread_start_us":7776,"threads_started":1}
I20260620 20:13:37.383177 20949 tablet_service.cc:1467] Processing CreateTablet for tablet 017f28afa7294822a70d03d6cceb79c2 (DEFAULT_TABLE table=test-table [id=32d0f9a9ef5c4d23a8cb1bdb23a96edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:37.384634 20949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 017f28afa7294822a70d03d6cceb79c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:37.390580 21293 tablet_bootstrap.cc:492] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: Bootstrap starting.
I20260620 20:13:37.393522 21293 tablet_bootstrap.cc:654] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:37.394632 21294 tablet_bootstrap.cc:492] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Bootstrap starting.
I20260620 20:13:37.395268 21293 log.cc:824] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:37.398048 21294 tablet_bootstrap.cc:654] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:37.398602 21293 tablet_bootstrap.cc:492] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: No bootstrap required, opened a new log
I20260620 20:13:37.398870 21293 ts_tablet_manager.cc:1397] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 20:13:37.399048 21296 tablet_bootstrap.cc:492] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: Bootstrap starting.
I20260620 20:13:37.399639 21294 log.cc:824] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:37.402555 21296 tablet_bootstrap.cc:654] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:37.403620 21294 tablet_bootstrap.cc:492] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: No bootstrap required, opened a new log
I20260620 20:13:37.403769 21296 log.cc:824] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:37.403885 21294 ts_tablet_manager.cc:1397] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 20:13:37.406850 21296 tablet_bootstrap.cc:492] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: No bootstrap required, opened a new log
I20260620 20:13:37.407116 21296 ts_tablet_manager.cc:1397] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 20:13:37.407814 21293 raft_consensus.cc:348] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:37.408320 21014 heartbeater.cc:503] Master 127.4.169.190:41517 was elected leader, sending a full tablet report...
I20260620 20:13:37.409650 21293 raft_consensus.cc:374] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:37.409768 21293 raft_consensus.cc:729] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913784e4458e4deb941873917f9ff593, State: Initialized, Role: FOLLOWER
I20260620 20:13:37.410430 21293 consensus_queue.cc:260] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:37.414469 21294 raft_consensus.cc:348] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:37.414914 21294 raft_consensus.cc:374] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:37.414999 21294 raft_consensus.cc:729] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e0bbe7e0e944aca2b7c1cb4471bf0a, State: Initialized, Role: FOLLOWER
I20260620 20:13:37.415616 21294 consensus_queue.cc:260] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:37.416442 21277 heartbeater.cc:503] Master 127.4.169.190:41517 was elected leader, sending a full tablet report...
I20260620 20:13:37.417068 21293 ts_tablet_manager.cc:1428] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: Time spent starting tablet: real 0.018s user 0.005s sys 0.010s
I20260620 20:13:37.418656 21146 heartbeater.cc:503] Master 127.4.169.190:41517 was elected leader, sending a full tablet report...
I20260620 20:13:37.419332 21294 ts_tablet_manager.cc:1428] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260620 20:13:37.420337 21296 raft_consensus.cc:348] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:37.420823 21296 raft_consensus.cc:374] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:37.420949 21296 raft_consensus.cc:729] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fead451fdd24d8987a458ff9d269a3c, State: Initialized, Role: FOLLOWER
I20260620 20:13:37.421597 21296 consensus_queue.cc:260] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:37.423460 21296 ts_tablet_manager.cc:1428] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260620 20:13:37.540164 21278 tablet.cc:2369] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:13:37.572304 21147 tablet.cc:2369] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:37.587250 21301 raft_consensus.cc:482] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:37.587622 21301 raft_consensus.cc:504] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:37.589342 21301 leader_election.cc:288] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38e0bbe7e0e944aca2b7c1cb4471bf0a (127.4.169.130:45559), 913784e4458e4deb941873917f9ff593 (127.4.169.131:45885)
I20260620 20:13:37.596313 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.589896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:37999 (local address 127.4.169.130:45559)
0620 20:13:37.590129 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.590138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.590875 (+ 737us) server_negotiation.cc:407] Connection header received
0620 20:13:37.591129 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.591133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.591191 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.591289 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:37.591912 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.592609 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.594597 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.595367 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.595549 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.595634 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.595913 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.596051 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:13:37.596326 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.589798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:57225 (local address 127.4.169.131:45885)
0620 20:13:37.590064 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.590073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.590862 (+ 789us) server_negotiation.cc:407] Connection header received
0620 20:13:37.591091 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.591096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.591155 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.591249 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:37.591900 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.592615 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.594589 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.595368 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.595577 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.595662 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.595956 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.596078 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:13:37.596488 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.589773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:45885 (local address 127.4.169.129:57225)
0620 20:13:37.590736 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:37.590760 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:37.590937 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:37.591273 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:37.591277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:37.591301 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:37.591756 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.591765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.592749 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.592756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.594435 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.594456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.595657 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.595663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.595805 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.595826 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:13:37.595930 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:13:37.596058 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":100,"thread_start_us":637,"threads_started":1}
I20260620 20:13:37.596711 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.589840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:45559 (local address 127.4.169.129:37999)
0620 20:13:37.590769 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:37.590784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:37.590981 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:37.591307 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:37.591311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:37.591333 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:37.591759 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.591768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.592744 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.592749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.594430 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.594452 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.595607 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.595613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.595759 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.595782 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:13:37.595889 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:13:37.596020 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":59,"spinlock_wait_cycles":34048,"thread_start_us":454,"threads_started":1}
I20260620 20:13:37.597482 21232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017f28afa7294822a70d03d6cceb79c2" candidate_uuid: "0fead451fdd24d8987a458ff9d269a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913784e4458e4deb941873917f9ff593" is_pre_election: true
I20260620 20:13:37.597644 21101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017f28afa7294822a70d03d6cceb79c2" candidate_uuid: "0fead451fdd24d8987a458ff9d269a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" is_pre_election: true
I20260620 20:13:37.598012 21232 raft_consensus.cc:2454] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fead451fdd24d8987a458ff9d269a3c in term 0.
I20260620 20:13:37.598183 21101 raft_consensus.cc:2454] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fead451fdd24d8987a458ff9d269a3c in term 0.
I20260620 20:13:37.598747 20905 leader_election.cc:302] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fead451fdd24d8987a458ff9d269a3c, 913784e4458e4deb941873917f9ff593; no voters:
I20260620 20:13:37.599161 21301 raft_consensus.cc:2781] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:37.599310 21301 raft_consensus.cc:482] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:37.599416 21301 raft_consensus.cc:3037] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:37.601706 21301 raft_consensus.cc:504] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:37.602296 21301 leader_election.cc:288] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [CANDIDATE]: Term 1 election: Requested vote from peers 38e0bbe7e0e944aca2b7c1cb4471bf0a (127.4.169.130:45559), 913784e4458e4deb941873917f9ff593 (127.4.169.131:45885)
I20260620 20:13:37.602821 21101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017f28afa7294822a70d03d6cceb79c2" candidate_uuid: "0fead451fdd24d8987a458ff9d269a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a"
I20260620 20:13:37.602856 21232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017f28afa7294822a70d03d6cceb79c2" candidate_uuid: "0fead451fdd24d8987a458ff9d269a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913784e4458e4deb941873917f9ff593"
I20260620 20:13:37.603037 21101 raft_consensus.cc:3037] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:37.603089 21232 raft_consensus.cc:3037] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:37.605321 21101 raft_consensus.cc:2454] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fead451fdd24d8987a458ff9d269a3c in term 1.
I20260620 20:13:37.605340 21232 raft_consensus.cc:2454] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fead451fdd24d8987a458ff9d269a3c in term 1.
I20260620 20:13:37.605808 20904 leader_election.cc:302] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fead451fdd24d8987a458ff9d269a3c, 38e0bbe7e0e944aca2b7c1cb4471bf0a; no voters:
I20260620 20:13:37.606158 21301 raft_consensus.cc:2781] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:37.606662 21301 raft_consensus.cc:686] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 1 LEADER]: Becoming Leader. State: Replica: 0fead451fdd24d8987a458ff9d269a3c, State: Running, Role: LEADER
I20260620 20:13:37.607219 21301 consensus_queue.cc:237] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } }
I20260620 20:13:37.611950 20829 catalog_manager.cc:5495] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fead451fdd24d8987a458ff9d269a3c (127.4.169.129). New cstate: current_term: 1 leader_uuid: "0fead451fdd24d8987a458ff9d269a3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fead451fdd24d8987a458ff9d269a3c" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39361 } health_report { overall_health: HEALTHY } } }
W20260620 20:13:37.622696 21015 tablet.cc:2369] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:37.683079 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.677164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.4.169.129:39361)
0620 20:13:37.677395 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.677401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.677448 (+ 47us) server_negotiation.cc:407] Connection header received
0620 20:13:37.677720 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.677724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.677819 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.677979 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:37.678348 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.679171 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.680597 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.681040 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.681185 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.681720 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:37.682277 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:37.682644 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.682713 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.682800 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:13:37.691643 21232 raft_consensus.cc:1261] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [term 1 FOLLOWER]: Refusing update from remote peer 0fead451fdd24d8987a458ff9d269a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:37.691643 21101 raft_consensus.cc:1261] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 1 FOLLOWER]: Refusing update from remote peer 0fead451fdd24d8987a458ff9d269a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:37.692997 21301 consensus_queue.cc:1035] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:37.693336 21304 consensus_queue.cc:1035] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "913784e4458e4deb941873917f9ff593" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:37.705549 21307 mvcc.cc:205] Tried to move back new op lower bound from 7299016366853500928 to 7299016366523146240. Current Snapshot: MvccSnapshot[applied={T|T < 7299016366853500928}]
I20260620 20:13:37.719483 20829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:37.719748 20829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:37.723307 20829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Sending DeleteTablet for 3 replicas of tablet 017f28afa7294822a70d03d6cceb79c2
I20260620 20:13:37.724613 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet 017f28afa7294822a70d03d6cceb79c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:37 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:41292
I20260620 20:13:37.724798 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 20866
I20260620 20:13:37.724746 21081 tablet_service.cc:1514] Processing DeleteTablet for tablet 017f28afa7294822a70d03d6cceb79c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:37 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:51798
I20260620 20:13:37.725577 20949 tablet_service.cc:1514] Processing DeleteTablet for tablet 017f28afa7294822a70d03d6cceb79c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:37 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:37006
I20260620 20:13:37.726431 21319 tablet_replica.cc:330] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: stopping tablet replica
I20260620 20:13:37.726910 21319 raft_consensus.cc:2229] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:37.727698 21319 raft_consensus.cc:2258] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:37.729926 21319 ts_tablet_manager.cc:1905] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:37.736296 21317 tablet_replica.cc:330] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: stopping tablet replica
I20260620 20:13:37.736881 21317 raft_consensus.cc:2229] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:37.737509 21319 ts_tablet_manager.cc:1918] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:37.737537 21317 raft_consensus.cc:2258] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:37.737762 21319 log.cc:1196] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/017f28afa7294822a70d03d6cceb79c2
I20260620 20:13:37.738608 21319 ts_tablet_manager.cc:1939] T 017f28afa7294822a70d03d6cceb79c2 P 0fead451fdd24d8987a458ff9d269a3c: Deleting consensus metadata
I20260620 20:13:37.740562 21318 tablet_replica.cc:330] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: stopping tablet replica
I20260620 20:13:37.740599 21317 ts_tablet_manager.cc:1905] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:37.740644 20815 catalog_manager.cc:4841] TS 0fead451fdd24d8987a458ff9d269a3c (127.4.169.129:39361): tablet 017f28afa7294822a70d03d6cceb79c2 (table test-table [id=32d0f9a9ef5c4d23a8cb1bdb23a96edf]) successfully deleted
I20260620 20:13:37.741149 21318 raft_consensus.cc:2229] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:37.741690 21318 raft_consensus.cc:2258] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:37.743891 21318 ts_tablet_manager.cc:1905] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:37.744853 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.725611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.4.169.129:39361)
0620 20:13:37.725826 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.725835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.726483 (+ 648us) server_negotiation.cc:407] Connection header received
0620 20:13:37.726546 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.726551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.726623 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.726721 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:37.728934 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.730005 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.732070 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.732774 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.732890 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.736607 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:37.736620 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:37.736638 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:37.736690 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:37.741221 (+ 4531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.742039 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.742047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.742050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.742224 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.742722 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.742728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.742731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.743179 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:37.743510 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.744190 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.744456 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:13:37.746328 21009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:37.748072 21317 ts_tablet_manager.cc:1918] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:37.748242 21317 log.cc:1196] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/017f28afa7294822a70d03d6cceb79c2
I20260620 20:13:37.748805 21317 ts_tablet_manager.cc:1939] T 017f28afa7294822a70d03d6cceb79c2 P 913784e4458e4deb941873917f9ff593: Deleting consensus metadata
I20260620 20:13:37.750711 20816 catalog_manager.cc:4841] TS 913784e4458e4deb941873917f9ff593 (127.4.169.131:45885): tablet 017f28afa7294822a70d03d6cceb79c2 (table test-table [id=32d0f9a9ef5c4d23a8cb1bdb23a96edf]) successfully deleted
I20260620 20:13:37.751976 21318 ts_tablet_manager.cc:1918] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:37.752141 21318 log.cc:1196] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/017f28afa7294822a70d03d6cceb79c2
I20260620 20:13:37.752588 21318 ts_tablet_manager.cc:1939] T 017f28afa7294822a70d03d6cceb79c2 P 38e0bbe7e0e944aca2b7c1cb4471bf0a: Deleting consensus metadata
I20260620 20:13:37.754052 20815 catalog_manager.cc:4841] TS 38e0bbe7e0e944aca2b7c1cb4471bf0a (127.4.169.130:45559): tablet 017f28afa7294822a70d03d6cceb79c2 (table test-table [id=32d0f9a9ef5c4d23a8cb1bdb23a96edf]) successfully deleted
I20260620 20:13:37.843465 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 20866
I20260620 20:13:37.860627 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 21018
I20260620 20:13:37.874730 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.861396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.4.169.130:45559)
0620 20:13:37.861575 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.861581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.861599 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:13:37.861766 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.861772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.861847 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.861947 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:37.863319 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.864066 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.865526 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.865895 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.865993 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.868631 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:37.868645 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:37.868659 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:37.868697 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:37.871756 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.872355 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.872361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.872364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.872494 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.872850 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.872856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.872860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.873279 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:37.873624 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.873995 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.874218 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:13:37.875438 21141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:37.965179 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 21018
I20260620 20:13:37.982137 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 21149
I20260620 20:13:37.995760 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.982931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.4.169.131:45885)
0620 20:13:37.983140 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.983147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.983167 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:13:37.983383 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.983388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.983455 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.983593 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:37.984940 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.985646 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.987105 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.987508 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.987646 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.990328 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:37.990343 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:37.990359 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:37.990432 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:37.992937 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.993672 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.993686 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.993688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.993821 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:37.994193 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:37.994200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:37.994203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:37.994607 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:37.994899 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.995208 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.995421 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:13:37.996356 21272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:38.085839 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 21149
I20260620 20:13:38.102276 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 20796
I20260620 20:13:38.103541 20859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:13:38.217113 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 20796
I20260620 20:13:58.239535 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41517
--webserver_interface=127.4.169.190
--webserver_port=36189
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:58.573861 21324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:58.574152 21324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:58.574311 21324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:58.581638 21324 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:13:58.581743 21324 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:13:58.581818 21324 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:13:58.581895 21324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:58.581990 21324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:58.582114 21324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:58.582191 21324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:58.582275 21324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:58.592617 21324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=36189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:58.593945 21324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:58.595359 21324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:58.607486 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:58.607465 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:58.607510 21332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:58.608253 21324 server_base.cc:1034] running on GCE node
I20260620 20:13:58.609062 21324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:58.610525 21324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:58.611778 21324 hybrid_clock.cc:648] HybridClock initialized: now 1781986438611711 us; error 55 us; skew 500 ppm
I20260620 20:13:58.612126 21324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:58 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986438, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:58.615293 21324 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:13:58.616828 21324 webserver.cc:460] Webserver started at http://127.4.169.190:36189/ using document root <none> and password file <none>
I20260620 20:13:58.617523 21324 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:58.617671 21324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:58.623231 21324 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 20:13:58.626024 21339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:58.627486 21324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:13:58.627686 21324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "49ef4c284b08475b8b12ee54a8d16b56"
format_stamp: "Formatted at 2026-06-20 20:13:12 on dist-test-slave-4bwq"
server_key: "caddc1fc2153f1e0ad506731da36194f"
server_key_iv: "6b83b89afbad86909657c63f906b41e5"
server_key_version: "encryptionkey@0"
I20260620 20:13:58.628667 21324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:58.644214 21324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:58.648710 21324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:58.649017 21324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:58.668673 21324 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:41517
I20260620 20:13:58.668706 21390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:41517 every 8 connection(s)
I20260620 20:13:58.670187 21324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:58.673542 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 21324
I20260620 20:13:58.674336 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:39361
--local_ip_for_outbound_sockets=127.4.169.129
--tserver_master_addrs=127.4.169.190:41517
--webserver_port=43657
--webserver_interface=127.4.169.129
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:13:58.677366 21391 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:13:58.680655 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Bootstrap starting.
I20260620 20:13:58.711776 21391 log.cc:824] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:58.732999 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:13:58.733743 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Bootstrap complete.
I20260620 20:13:58.749086 21391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:58.750432 21391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef4c284b08475b8b12ee54a8d16b56, State: Initialized, Role: FOLLOWER
I20260620 20:13:58.751180 21391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:58.751471 21391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:58.751592 21391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:58.751776 21391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:13:58.757328 21391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:58.757974 21391 leader_election.cc:302] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49ef4c284b08475b8b12ee54a8d16b56; no voters:
I20260620 20:13:58.758701 21391 leader_election.cc:288] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:13:58.758872 21395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:13:58.760310 21395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [term 3 LEADER]: Becoming Leader. State: Replica: 49ef4c284b08475b8b12ee54a8d16b56, State: Running, Role: LEADER
I20260620 20:13:58.760810 21391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:58.761039 21395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } }
I20260620 20:13:58.763731 21396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "49ef4c284b08475b8b12ee54a8d16b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } } }
I20260620 20:13:58.763754 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ef4c284b08475b8b12ee54a8d16b56. Latest consensus state: current_term: 3 leader_uuid: "49ef4c284b08475b8b12ee54a8d16b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef4c284b08475b8b12ee54a8d16b56" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41517 } } }
I20260620 20:13:58.764161 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:58.764281 21396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:58.779771 21401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:58.788419 21401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:58.789896 21401 catalog_manager.cc:1259] Loaded cluster ID: 7d38f7d7215b49bd8f65b47bfd49723d
I20260620 20:13:58.790005 21401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:58.794075 21401 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:58.795800 21401 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Loaded TSK: 0
I20260620 20:13:58.795940 21401 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49ef4c284b08475b8b12ee54a8d16b56: Loaded TSK: 1
F20260620 20:13:58.797049 21401 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9e3b9c99ad google::LogMessage::Fail() at ??:0
@ 0x7f9e3b9cc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9e3b9c94de google::LogMessage::Flush() at ??:0
@ 0x7f9e3b9caa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9e3f72d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9e3f71c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9e513fc1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9e513fb783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9e514872a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9e514870c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f9e514d15e9 std::function<>::operator()() at ??:0
@ 0x7f9e513ffcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9e513fdc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9e51485610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9e51485439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9e3e615345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9e3e61891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9e3e618799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9e3e5f3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9e418cd6db start_thread at ??:0
@ 0x7f9e3a86361f clone at ??:0
W20260620 20:13:59.097790 21393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:59.098127 21393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:59.098287 21393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:59.105866 21393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:59.105981 21393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:59.106199 21393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:13:59.116498 21393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:39361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=43657
--tserver_master_addrs=127.4.169.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:59.117825 21393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:59.119223 21393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:59.132314 21438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:59.132344 21441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:59.132406 21439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:59.133229 21393 server_base.cc:1034] running on GCE node
I20260620 20:13:59.133859 21393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:59.134877 21393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:59.136106 21393 hybrid_clock.cc:648] HybridClock initialized: now 1781986439136057 us; error 44 us; skew 500 ppm
I20260620 20:13:59.136425 21393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:59 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986439, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:59.139544 21393 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:13:59.140908 21393 webserver.cc:460] Webserver started at http://127.4.169.129:43657/ using document root <none> and password file <none>
I20260620 20:13:59.141551 21393 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:59.141681 21393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:59.147018 21393 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:13:59.149674 21448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:59.150992 21393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:59.151170 21393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "0fead451fdd24d8987a458ff9d269a3c"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "98f261f0c8311c408bc5afe0009b7191"
server_key_iv: "c2c66c0be5883eb530e01c3e5a171c29"
server_key_version: "encryptionkey@0"
I20260620 20:13:59.152128 21393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:59.169932 21393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:59.174152 21393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:59.174496 21393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:59.175557 21393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:59.176882 21393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:59.176998 21393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:59.177132 21393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:59.177239 21393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:59.210904 21393 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:39361
I20260620 20:13:59.210994 21560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:39361 every 8 connection(s)
I20260620 20:13:59.212389 21393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:59.213212 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 21393
I20260620 20:13:59.214033 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:45559
--local_ip_for_outbound_sockets=127.4.169.130
--tserver_master_addrs=127.4.169.190:41517
--webserver_port=39545
--webserver_interface=127.4.169.130
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:13:59.217932 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:59.215631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:33647)
0620 20:13:59.216480 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:59.216756 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":316,"threads_started":1}
W20260620 20:13:59.218631 21561 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:41517 (0 consecutive failures): Network error: Failed to ping master at 127.4.169.190:41517: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
I20260620 20:13:59.219586 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:59.219142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:50975)
0620 20:13:59.219335 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:59.219432 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 20:13:59.220556 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:59.220131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:47173)
0620 20:13:59.220303 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:59.220401 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:13:59.220850 21561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:13:59.542320 21564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:59.542675 21564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:59.542847 21564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:59.550127 21564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:59.550282 21564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:59.550514 21564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:13:59.560384 21564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:45559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=39545
--tserver_master_addrs=127.4.169.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:13:59.561694 21564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:59.563131 21564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:59.576208 21569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:59.576306 21572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:59.576282 21570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:59.577028 21564 server_base.cc:1034] running on GCE node
I20260620 20:13:59.577572 21564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:59.578598 21564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:59.579824 21564 hybrid_clock.cc:648] HybridClock initialized: now 1781986439579779 us; error 56 us; skew 500 ppm
I20260620 20:13:59.580135 21564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:13:59 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986439, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:59.583257 21564 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:13:59.584667 21564 webserver.cc:460] Webserver started at http://127.4.169.130:39545/ using document root <none> and password file <none>
I20260620 20:13:59.585343 21564 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:59.585490 21564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:59.590972 21564 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:13:59.593786 21579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:59.595086 21564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:59.595263 21564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "38e0bbe7e0e944aca2b7c1cb4471bf0a"
format_stamp: "Formatted at 2026-06-20 20:13:13 on dist-test-slave-4bwq"
server_key: "2f1c05ef17754f6b491e29e0ba3f3b89"
server_key_iv: "86eb1873d6b2d0c348ab6727721dcc76"
server_key_version: "encryptionkey@0"
I20260620 20:13:59.596259 21564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:59.613981 21564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:59.618551 21564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:59.618840 21564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:59.619870 21564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:59.621289 21564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:59.621417 21564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:59.621562 21564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:59.621652 21564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:59.656594 21564 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:45559
I20260620 20:13:59.656689 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:45559 every 8 connection(s)
I20260620 20:13:59.658113 21564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:59.659706 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 21564
I20260620 20:13:59.660502 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:45885
--local_ip_for_outbound_sockets=127.4.169.131
--tserver_master_addrs=127.4.169.190:41517
--webserver_port=33117
--webserver_interface=127.4.169.131
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:13:59.664026 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:59.661649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:40845)
0620 20:13:59.662538 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:59.662824 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":321,"threads_started":1}
W20260620 20:13:59.664772 21692 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:41517 (0 consecutive failures): Network error: Failed to ping master at 127.4.169.190:41517: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
I20260620 20:13:59.665889 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:59.665406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:49263)
0620 20:13:59.665617 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:59.665726 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 20:13:59.666965 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:59.666568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:42781)
0620 20:13:59.666721 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:59.666826 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:13:59.667260 21692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:13:59.986464 21695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:59.986737 21695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:59.986860 21695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:59.994230 21695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:59.994334 21695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:59.994506 21695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:14:00.004628 21695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:45885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=33117
--tserver_master_addrs=127.4.169.190:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:00.005918 21695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:00.007396 21695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:00.021433 21701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:00.021502 21703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:00.021481 21700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:00.022850 21695 server_base.cc:1034] running on GCE node
I20260620 20:14:00.023550 21695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:00.024550 21695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:00.025803 21695 hybrid_clock.cc:648] HybridClock initialized: now 1781986440025758 us; error 41 us; skew 500 ppm
I20260620 20:14:00.026170 21695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:00 dist-test-slave-4bwq krb5kdc[20245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986440, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:00.029527 21695 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:14:00.031148 21695 webserver.cc:460] Webserver started at http://127.4.169.131:33117/ using document root <none> and password file <none>
I20260620 20:14:00.031862 21695 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:00.032012 21695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:00.037569 21695 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:14:00.040289 21710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:00.041558 21695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:14:00.041728 21695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "913784e4458e4deb941873917f9ff593"
format_stamp: "Formatted at 2026-06-20 20:13:14 on dist-test-slave-4bwq"
server_key: "9f534b9e49171e32c0ef947dbe98882e"
server_key_iv: "a897bc9c5974968be2888d8724e6c19b"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.042748 21695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:00.073365 21695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:00.077654 21695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:00.077955 21695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:00.079018 21695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:00.080490 21695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:00.080610 21695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:00.080724 21695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:00.080837 21695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:00.115237 21695 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:45885
I20260620 20:14:00.115301 21822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:45885 every 8 connection(s)
I20260620 20:14:00.116766 21695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:00.121661 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.119852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:35513)
0620 20:14:00.120586 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:00.120813 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":300,"threads_started":1}
W20260620 20:14:00.122429 21823 heartbeater.cc:644] Failed to heartbeat to 127.4.169.190:41517 (0 consecutive failures): Network error: Failed to ping master at 127.4.169.190:41517: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
I20260620 20:14:00.123337 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.122937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:53413)
0620 20:14:00.123087 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:00.123188 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 20:14:00.124168 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.123831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:33471)
0620 20:14:00.123972 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:00.124058 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:14:00.124403 21823 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:14:00.126588 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 21695
I20260620 20:14:00.222131 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.221358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:54985)
0620 20:14:00.221782 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:00.221885 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":206,"threads_started":1}
I20260620 20:14:00.668550 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.667793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:50711)
0620 20:14:00.668218 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:00.668323 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260620 20:14:01.125955 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.125008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:40643)
0620 20:14:01.125565 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.125698 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":222,"threads_started":1}
I20260620 20:14:01.223896 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.223169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:35327)
0620 20:14:01.223640 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.223728 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":242,"threads_started":1}
I20260620 20:14:01.670060 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.669378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:47355)
0620 20:14:01.669822 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.669899 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":217,"threads_started":1}
I20260620 20:14:02.127595 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.126901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:59973)
0620 20:14:02.127325 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.127404 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
I20260620 20:14:02.225411 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.224779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:46555)
0620 20:14:02.225173 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.225249 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":212,"threads_started":1}
I20260620 20:14:02.671597 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.670925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:49455)
0620 20:14:02.671336 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.671413 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260620 20:14:03.129256 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:03.128535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:40553)
0620 20:14:03.129001 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:03.129087 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":256,"threads_started":1}
I20260620 20:14:03.227267 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:03.226543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:39589)
0620 20:14:03.226987 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:03.227074 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":208,"threads_started":1}
I20260620 20:14:03.673409 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:03.672554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:59027)
0620 20:14:03.673123 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:03.673212 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
I20260620 20:14:04.131008 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:04.130190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:56155)
0620 20:14:04.130738 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:04.130827 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":312,"threads_started":1}
I20260620 20:14:04.228885 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:04.228171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:58753)
0620 20:14:04.228642 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:04.228721 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":228,"threads_started":1}
I20260620 20:14:04.675181 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:04.674451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:43469)
0620 20:14:04.674930 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:04.675011 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":242,"threads_started":1}
I20260620 20:14:05.132699 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.132016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:58679)
0620 20:14:05.132451 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:05.132531 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":215,"threads_started":1}
I20260620 20:14:05.230537 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.229756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:36943)
0620 20:14:05.230207 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:05.230299 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260620 20:14:05.677229 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.676194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:35131)
0620 20:14:05.676847 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:05.676982 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":327,"threads_started":1}
I20260620 20:14:06.134667 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.133660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:34585)
0620 20:14:06.134279 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.134403 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":349,"threads_started":1}
I20260620 20:14:06.232133 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.231361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:52843)
0620 20:14:06.231830 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.231912 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":209,"threads_started":1}
I20260620 20:14:06.678957 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.678136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:43109)
0620 20:14:06.678657 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.678739 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":260,"threads_started":1}
I20260620 20:14:07.136406 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:07.135629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:38863)
0620 20:14:07.136118 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:07.136201 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":208,"threads_started":1}
I20260620 20:14:07.233819 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:07.233076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:42931)
0620 20:14:07.233551 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:07.233646 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":233,"threads_started":1}
I20260620 20:14:07.680626 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:07.679906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:36411)
0620 20:14:07.680364 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:07.680452 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260620 20:14:08.138120 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:08.137363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:59145)
0620 20:14:08.137832 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:08.137928 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260620 20:14:08.235415 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:08.234687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:48959)
0620 20:14:08.235158 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:08.235240 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":220,"threads_started":1}
I20260620 20:14:08.682318 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:08.681545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:42053)
0620 20:14:08.682050 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:08.682133 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":244,"threads_started":1}
I20260620 20:14:09.140020 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:09.139190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:54365)
0620 20:14:09.139708 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:09.139835 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
I20260620 20:14:09.237185 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:09.236482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:36607)
0620 20:14:09.236919 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:09.237025 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":242,"threads_started":1}
I20260620 20:14:09.684031 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:09.683341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:36345)
0620 20:14:09.683774 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:09.683856 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
I20260620 20:14:10.141716 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:10.140973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:43191)
0620 20:14:10.141437 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:10.141542 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":251,"threads_started":1}
I20260620 20:14:10.238915 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:10.238136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:60853)
0620 20:14:10.238638 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:10.238734 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
I20260620 20:14:10.685835 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:10.685081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:50809)
0620 20:14:10.685555 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:10.685656 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":230,"threads_started":1}
I20260620 20:14:11.143374 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:11.142674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:54345)
0620 20:14:11.143105 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:11.143196 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":211,"threads_started":1}
I20260620 20:14:11.240700 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:11.239913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:59931)
0620 20:14:11.240417 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:11.240531 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":275,"threads_started":1}
I20260620 20:14:11.687691 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:11.686839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:57539)
0620 20:14:11.687444 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:11.687532 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":412,"threads_started":1}
I20260620 20:14:12.145117 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:12.144414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:60919)
0620 20:14:12.144869 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:12.144953 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":236,"threads_started":1}
I20260620 20:14:12.242635 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:12.241731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:40765)
0620 20:14:12.242268 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:12.242372 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":279,"threads_started":1}
I20260620 20:14:12.689496 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:12.688723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:43633)
0620 20:14:12.689211 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:12.689309 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":250,"threads_started":1}
I20260620 20:14:13.147045 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:13.146157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:56327)
0620 20:14:13.146728 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:13.146848 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":314,"threads_started":1}
I20260620 20:14:13.244179 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:13.243503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:38475)
0620 20:14:13.243908 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:13.243991 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":205,"threads_started":1}
I20260620 20:14:13.691154 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:13.690459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:60149)
0620 20:14:13.690873 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:13.690968 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":207,"threads_started":1}
I20260620 20:14:14.148581 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:14.147867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:56093)
0620 20:14:14.148321 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:14.148405 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
I20260620 20:14:14.245959 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:14.245228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.129:56771)
0620 20:14:14.245659 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:14.245748 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20260620 20:14:14.693078 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:14.692263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.130:38895)
0620 20:14:14.692744 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:14.692838 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":261,"threads_started":1}
I20260620 20:14:15.131384 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 21393
I20260620 20:14:15.150542 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:15.149715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41517 (local address 127.4.169.131:46351)
0620 20:14:15.150189 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:15.150276 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.190:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":256,"threads_started":1}
I20260620 20:14:15.155709 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:15.132457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.4.169.129:39361)
0620 20:14:15.133089 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:15.133109 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:15.133160 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:14:15.135006 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:15.135016 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:15.135439 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:15.135748 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:15.137141 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.138383 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.139670 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.140344 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.140477 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:15.148749 (+ 8272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:15.148758 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:15.148776 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:15.148821 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:15.152179 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:15.152927 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:15.152934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:15.152941 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:15.153123 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:15.153528 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:15.153535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:15.153542 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:15.154083 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:15.154389 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:15.154909 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 20:14:15.155402 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":431,"threads_started":1}
I20260620 20:14:15.157130 21556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:15.244701 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 21393
I20260620 20:14:15.258447 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 21564
I20260620 20:14:15.276021 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:15.259402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.4.169.130:45559)
0620 20:14:15.259955 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:15.259969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:15.260005 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:14:15.261556 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:15.261565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:15.261998 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:15.262343 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:15.263808 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.265031 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.266424 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.267092 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.267291 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:15.269912 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:15.269919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:15.269935 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:15.269971 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:15.272793 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:15.273484 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:15.273492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:15.273498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:15.273658 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:15.274035 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:15.274040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:15.274043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:15.274608 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:15.274916 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:15.275335 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 20:14:15.275732 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":260,"threads_started":1}
I20260620 20:14:15.277391 21687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:15.361356 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 21564
I20260620 20:14:15.378983 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 21695
I20260620 20:14:15.396503 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:15.379970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.4.169.131:45885)
0620 20:14:15.380568 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:15.380582 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:15.380623 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:14:15.382280 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:15.382289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:15.382805 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:15.383118 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:15.384473 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.385672 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.387022 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.387670 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.387799 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:15.390325 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:15.390332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:15.390346 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:15.390472 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:15.393417 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:15.394049 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:15.394055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:15.394062 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:15.394222 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:15.394645 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:15.394650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:15.394652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:15.395200 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:15.395456 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:15.395764 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 20:14:15.396169 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":261,"threads_started":1}
I20260620 20:14:15.397912 21818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:15.482278 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 21695
2026-06-20T20:14:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64418 ms)
[----------] 30 tests from SecurityITest (302537 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:15 dist-test-slave-4bwq krb5kdc[21880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:15 dist-test-slave-4bwq krb5kdc[21880](info): set up 2 sockets
Jun 20 20:14:15 dist-test-slave-4bwq krb5kdc[21880](info): commencing operation
krb5kdc: starting...
W20260620 20:14:16.598846 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:16 dist-test-slave-4bwq krb5kdc[21880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:14:16.849336 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34473
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:17.182581 21893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:17.182927 21893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:17.183089 21893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:17.190340 21893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:17.190557 21893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:17.190653 21893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:17.190757 21893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:17.190826 21893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:17.200737 21893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:17.202088 21893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:17.203433 21893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:17.215209 21898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:17.215137 21901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:17.215174 21899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:17.216354 21893 server_base.cc:1034] running on GCE node
I20260620 20:14:17.217144 21893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:17.218668 21893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:17.219906 21893 hybrid_clock.cc:648] HybridClock initialized: now 1781986457219862 us; error 51 us; skew 500 ppm
I20260620 20:14:17.220288 21893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:17 dist-test-slave-4bwq krb5kdc[21880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:17.223660 21893 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:14:17.225307 21893 webserver.cc:460] Webserver started at http://127.4.169.190:32969/ using document root <none> and password file <none>
I20260620 20:14:17.226060 21893 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:17.226212 21893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:17.226593 21893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:17.229708 21893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "3593bdaad5c04b04a77714615d102cbc"
format_stamp: "Formatted at 2026-06-20 20:14:17 on dist-test-slave-4bwq"
server_key: "c20cc53e51b49b0e16c28602982daf28"
server_key_iv: "e5ee40e65744b69427e89746aa7b84a6"
server_key_version: "encryptionkey@0"
I20260620 20:14:17.230638 21893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "3593bdaad5c04b04a77714615d102cbc"
format_stamp: "Formatted at 2026-06-20 20:14:17 on dist-test-slave-4bwq"
server_key: "c20cc53e51b49b0e16c28602982daf28"
server_key_iv: "e5ee40e65744b69427e89746aa7b84a6"
server_key_version: "encryptionkey@0"
I20260620 20:14:17.235924 21893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:14:17.239756 21908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:17.241242 21893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260620 20:14:17.241448 21893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "3593bdaad5c04b04a77714615d102cbc"
format_stamp: "Formatted at 2026-06-20 20:14:17 on dist-test-slave-4bwq"
server_key: "c20cc53e51b49b0e16c28602982daf28"
server_key_iv: "e5ee40e65744b69427e89746aa7b84a6"
server_key_version: "encryptionkey@0"
I20260620 20:14:17.241667 21893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:17.261538 21893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:17.266093 21893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:17.266415 21893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:17.285188 21893 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:34473
I20260620 20:14:17.285220 21959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:34473 every 8 connection(s)
I20260620 20:14:17.286645 21893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:17.290432 21960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:17.292501 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 21893
I20260620 20:14:17.292739 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:17.293592 4774 external_mini_cluster.cc:1428] Setting key e826ef147b9eb1243ce8ac28b2078502
I20260620 20:14:17.303435 21960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc: Bootstrap starting.
I20260620 20:14:17.307360 21960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:17.308624 21960 log.cc:824] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:14:17 dist-test-slave-4bwq krb5kdc[21880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:17.311403 21960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc: No bootstrap required, opened a new log
I20260620 20:14:17.316972 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:17.295739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.4.169.190:34473)
0620 20:14:17.296572 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:17.296584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:17.296626 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:14:17.298326 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:17.298349 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:17.298809 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:17.299124 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:17.300894 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.302221 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.304136 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.304765 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.304899 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:17.309723 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:17.309731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:17.309746 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:17.309780 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:17.312835 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:17.313581 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:17.313587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:17.313595 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:17.313763 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:17.314177 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:17.314186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:17.314190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:17.314787 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:17.315104 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:17.315562 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 20:14:17.315990 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":397,"threads_started":1}
I20260620 20:14:17.319718 21960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593bdaad5c04b04a77714615d102cbc" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34473 } }
I20260620 20:14:17.320176 21960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:17.320303 21960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3593bdaad5c04b04a77714615d102cbc, State: Initialized, Role: FOLLOWER
I20260620 20:14:17.320835 21960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593bdaad5c04b04a77714615d102cbc" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34473 } }
I20260620 20:14:17.321076 21960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:17.321202 21960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:17.321389 21960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:17.323755 21960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593bdaad5c04b04a77714615d102cbc" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34473 } }
I20260620 20:14:17.324234 21960 leader_election.cc:302] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3593bdaad5c04b04a77714615d102cbc; no voters:
I20260620 20:14:17.324913 21960 leader_election.cc:288] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:17.325040 21965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:17.325755 21965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [term 1 LEADER]: Becoming Leader. State: Replica: 3593bdaad5c04b04a77714615d102cbc, State: Running, Role: LEADER
I20260620 20:14:17.326398 21965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593bdaad5c04b04a77714615d102cbc" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34473 } }
I20260620 20:14:17.327256 21960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:17.329368 21966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3593bdaad5c04b04a77714615d102cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593bdaad5c04b04a77714615d102cbc" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34473 } } }
I20260620 20:14:17.329748 21966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:17.334682 21972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:17.334473 21967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3593bdaad5c04b04a77714615d102cbc. Latest consensus state: current_term: 1 leader_uuid: "3593bdaad5c04b04a77714615d102cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3593bdaad5c04b04a77714615d102cbc" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34473 } } }
I20260620 20:14:17.335016 21967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:17.340287 21972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:17.349310 21972 catalog_manager.cc:1347] Generated new cluster ID: 9f32800b6a214e36bfa30946b9bd6f77
I20260620 20:14:17.349434 21972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:17.374207 21972 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:17.375587 21972 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:17.386947 21972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc: Generated new TSK 0
I20260620 20:14:17.387830 21972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:14:17.454097 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34473
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:17.792824 21986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:17.793098 21986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:17.793267 21986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:17.800493 21986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:17.800602 21986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:17.800806 21986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:14:17.810637 21986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:17.811867 21986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:17.813156 21986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:17.826066 21991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:17.826187 21994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:17.826264 21992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:17.827340 21986 server_base.cc:1034] running on GCE node
I20260620 20:14:17.828012 21986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:17.828990 21986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:17.830214 21986 hybrid_clock.cc:648] HybridClock initialized: now 1781986457830174 us; error 43 us; skew 500 ppm
I20260620 20:14:17.830585 21986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:17 dist-test-slave-4bwq krb5kdc[21880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:17.833840 21986 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:14:17.835600 21986 webserver.cc:460] Webserver started at http://127.4.169.129:44671/ using document root <none> and password file <none>
I20260620 20:14:17.836311 21986 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:17.836457 21986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:17.836791 21986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:17.839757 21986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "ea1aed67780f4abfa3384fa12159751f"
format_stamp: "Formatted at 2026-06-20 20:14:17 on dist-test-slave-4bwq"
server_key: "04ed8e09cb22ac74dde4ea4738dcc2d0"
server_key_iv: "10fa8ef90865912f3507a6ff8351f950"
server_key_version: "encryptionkey@0"
I20260620 20:14:17.840535 21986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "ea1aed67780f4abfa3384fa12159751f"
format_stamp: "Formatted at 2026-06-20 20:14:17 on dist-test-slave-4bwq"
server_key: "04ed8e09cb22ac74dde4ea4738dcc2d0"
server_key_iv: "10fa8ef90865912f3507a6ff8351f950"
server_key_version: "encryptionkey@0"
I20260620 20:14:17.845688 21986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 20:14:17.849134 22001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:17.850358 21986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:17.850579 21986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "ea1aed67780f4abfa3384fa12159751f"
format_stamp: "Formatted at 2026-06-20 20:14:17 on dist-test-slave-4bwq"
server_key: "04ed8e09cb22ac74dde4ea4738dcc2d0"
server_key_iv: "10fa8ef90865912f3507a6ff8351f950"
server_key_version: "encryptionkey@0"
I20260620 20:14:17.850776 21986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:17.876822 21986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:17.881863 21986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:17.882203 21986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:17.883277 21986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:17.884624 21986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:17.884694 21986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:17.884866 21986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:17.884932 21986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:17.916419 21986 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:39505
I20260620 20:14:17.916458 22113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:39505 every 8 connection(s)
I20260620 20:14:17.917717 21986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:17.919131 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 21986
I20260620 20:14:17.919325 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:17.920136 4774 external_mini_cluster.cc:1428] Setting key 2ec7a423e108865ef7cec06d12f6e8fa
Jun 20 20:14:17 dist-test-slave-4bwq krb5kdc[21880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:17.941056 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:17.921473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:55349 (local address 127.4.169.190:34473)
0620 20:14:17.922699 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:17.922705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:17.922727 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:14:17.924338 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:17.924346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:17.924450 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:17.924570 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:17.926093 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.927381 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.929263 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.930004 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.930159 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:17.933784 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:17.933793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:17.933801 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:17.933847 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:17.937647 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:17.938367 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:17.938375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:17.938382 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:17.938563 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:17.938978 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:17.938986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:17.938992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:17.939387 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:17.939571 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:17.940273 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 20:14:17.940584 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":1114,"threads_started":1}
I20260620 20:14:17.941694 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:17.921171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34473 (local address 127.4.169.129:55349)
0620 20:14:17.922176 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:17.922248 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:17.924054 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:17.924774 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:17.924783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:17.925315 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:17.925911 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:17.925930 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.927657 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.927661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:17.929115 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:17.929127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.930219 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.930223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:17.930476 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:17.931192 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:17.931215 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:17.933558 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:17.937791 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:17.937798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:17.937812 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:17.938241 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:17.938679 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:17.938683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:17.938685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:17.938865 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:17.939553 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:17.939559 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:17.939877 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 20:14:17.940111 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 20:14:17.940537 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":376,"threads_started":1}
I20260620 20:14:17.943405 22114 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34473
I20260620 20:14:17.943727 22114 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:17.944412 22114 heartbeater.cc:511] Master 127.4.169.190:34473 requested a full tablet report, sending...
I20260620 20:14:17.946758 21925 ts_manager.cc:194] Registered new tserver with Master: ea1aed67780f4abfa3384fa12159751f (127.4.169.129:39505)
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:14:17.948307 21925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:55349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:14:17.978073 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34473
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:18.302588 22120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:18.302858 22120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:18.303056 22120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:18.310349 22120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:18.310552 22120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:18.310751 22120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:14:18.320552 22120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:18.321867 22120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:18.323252 22120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:18.337033 22126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:18.337031 22125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:18.337293 22128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:18.337872 22120 server_base.cc:1034] running on GCE node
I20260620 20:14:18.338537 22120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:18.339519 22120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:18.340785 22120 hybrid_clock.cc:648] HybridClock initialized: now 1781986458340732 us; error 45 us; skew 500 ppm
I20260620 20:14:18.341156 22120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:18 dist-test-slave-4bwq krb5kdc[21880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:18.344630 22120 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:14:18.346215 22120 webserver.cc:460] Webserver started at http://127.4.169.130:36765/ using document root <none> and password file <none>
I20260620 20:14:18.347049 22120 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:18.347214 22120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:18.347551 22120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:18.350701 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "44dfcf0aec86419fa4520b2d9e554a7f"
format_stamp: "Formatted at 2026-06-20 20:14:18 on dist-test-slave-4bwq"
server_key: "c17860ebb7100c1c05daa07e5e36a14a"
server_key_iv: "0a7c482344d3be968b4e382e87fd1b65"
server_key_version: "encryptionkey@0"
I20260620 20:14:18.351536 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "44dfcf0aec86419fa4520b2d9e554a7f"
format_stamp: "Formatted at 2026-06-20 20:14:18 on dist-test-slave-4bwq"
server_key: "c17860ebb7100c1c05daa07e5e36a14a"
server_key_iv: "0a7c482344d3be968b4e382e87fd1b65"
server_key_version: "encryptionkey@0"
I20260620 20:14:18.357113 22120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:14:18.360759 22135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:18.362192 22120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 20:14:18.362435 22120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "44dfcf0aec86419fa4520b2d9e554a7f"
format_stamp: "Formatted at 2026-06-20 20:14:18 on dist-test-slave-4bwq"
server_key: "c17860ebb7100c1c05daa07e5e36a14a"
server_key_iv: "0a7c482344d3be968b4e382e87fd1b65"
server_key_version: "encryptionkey@0"
I20260620 20:14:18.362681 22120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:18.403206 22120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:18.407894 22120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:18.408249 22120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:18.409278 22120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:18.410789 22120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:18.410895 22120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:18.411051 22120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:18.411126 22120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:18.445257 22120 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:37023
I20260620 20:14:18.445278 22247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:37023 every 8 connection(s)
I20260620 20:14:18.446821 22120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:18.451470 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 22120
I20260620 20:14:18.451716 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:18.452327 4774 external_mini_cluster.cc:1428] Setting key eb524ac19d3a26362ff08a54741c8b60
Jun 20 20:14:18 dist-test-slave-4bwq krb5kdc[21880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:18.472470 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:18.449488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:40051 (local address 127.4.169.190:34473)
0620 20:14:18.449940 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:18.449945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:18.451056 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 20:14:18.453762 (+ 2706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:18.453766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:18.453845 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:18.454003 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:18.456377 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.457972 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.459973 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.460755 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.460929 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.465295 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:18.465302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:18.465311 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:18.465363 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:18.468510 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:18.469630 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:18.469635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:18.469637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:18.469811 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:18.470349 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:18.470353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:18.470355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:18.470734 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:18.470876 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:18.471884 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0620 20:14:18.472163 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":216,"threads_started":1}
I20260620 20:14:18.474265 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:18.450101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34473 (local address 127.4.169.130:40051)
0620 20:14:18.450855 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:18.450898 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:18.453392 (+ 2494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:18.454516 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:18.454528 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:18.455409 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:18.456145 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.456179 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.458332 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.458338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.459786 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.459800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.461067 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.461074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.461328 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.462169 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:18.462201 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:18.464956 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:18.468676 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:18.468689 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:18.468711 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:18.469439 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:18.469933 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:18.469941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:18.469947 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:18.470219 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:18.470880 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:18.470892 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:18.471408 (+ 516us) client_negotiation.cc:769] Sending connection context
0620 20:14:18.471745 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 20:14:18.472387 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":350,"threads_started":1}
I20260620 20:14:18.476078 22248 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34473
I20260620 20:14:18.476439 22248 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:18.477260 22248 heartbeater.cc:511] Master 127.4.169.190:34473 requested a full tablet report, sending...
I20260620 20:14:18.479094 21925 ts_manager.cc:194] Registered new tserver with Master: 44dfcf0aec86419fa4520b2d9e554a7f (127.4.169.130:37023)
I20260620 20:14:18.479965 21925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:40051
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:14:18.514590 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34473
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:18.850267 22254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:18.850633 22254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:18.850805 22254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:18.858564 22254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:18.858719 22254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:18.858902 22254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:14:18.869217 22254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:18.870632 22254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:18.871969 22254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:18.885114 22262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:18.885087 22260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:18.885174 22259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:18.886005 22254 server_base.cc:1034] running on GCE node
I20260620 20:14:18.886672 22254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:18.887609 22254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:18.888811 22254 hybrid_clock.cc:648] HybridClock initialized: now 1781986458888747 us; error 62 us; skew 500 ppm
I20260620 20:14:18.889144 22254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:18 dist-test-slave-4bwq krb5kdc[21880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:18.892385 22254 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:14:18.893909 22254 webserver.cc:460] Webserver started at http://127.4.169.131:46257/ using document root <none> and password file <none>
I20260620 20:14:18.894692 22254 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:18.894835 22254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:18.895164 22254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:18.898124 22254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "6adf945b7a6448aa9d3212df10f5161f"
format_stamp: "Formatted at 2026-06-20 20:14:18 on dist-test-slave-4bwq"
server_key: "e25546ffac6074c8f7df8ead697a6364"
server_key_iv: "8f7eb00e5bda0f56ffc29e5787bd321c"
server_key_version: "encryptionkey@0"
I20260620 20:14:18.898989 22254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "6adf945b7a6448aa9d3212df10f5161f"
format_stamp: "Formatted at 2026-06-20 20:14:18 on dist-test-slave-4bwq"
server_key: "e25546ffac6074c8f7df8ead697a6364"
server_key_iv: "8f7eb00e5bda0f56ffc29e5787bd321c"
server_key_version: "encryptionkey@0"
I20260620 20:14:18.904094 22254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:14:18.907518 22269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:18.908757 22254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:14:18.908926 22254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "6adf945b7a6448aa9d3212df10f5161f"
format_stamp: "Formatted at 2026-06-20 20:14:18 on dist-test-slave-4bwq"
server_key: "e25546ffac6074c8f7df8ead697a6364"
server_key_iv: "8f7eb00e5bda0f56ffc29e5787bd321c"
server_key_version: "encryptionkey@0"
I20260620 20:14:18.909142 22254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:18.937028 22254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:18.941210 22254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:18.941457 22254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:18.942466 22254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:18.943790 22254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:18.943863 22254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:18.943948 22254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:18.943990 22254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:18.951508 22114 heartbeater.cc:503] Master 127.4.169.190:34473 was elected leader, sending a full tablet report...
I20260620 20:14:18.978538 22254 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:43359
I20260620 20:14:18.978626 22381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:43359 every 8 connection(s)
I20260620 20:14:18.979853 22254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:18.989935 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 22254
I20260620 20:14:18.990178 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:18.990712 4774 external_mini_cluster.cc:1428] Setting key c87f6cd5864a5ee2ddf5a4874350494e
Jun 20 20:14:18 dist-test-slave-4bwq krb5kdc[21880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:19.000038 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:18.982627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:43551 (local address 127.4.169.190:34473)
0620 20:14:18.983333 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:18.983337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:18.984201 (+ 864us) server_negotiation.cc:407] Connection header received
0620 20:14:18.986060 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:18.986066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:18.986153 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:18.986286 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:18.987825 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.988596 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.990134 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.990698 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.990836 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.994713 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:18.994718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:18.994721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:18.994753 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:18.997407 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:18.998047 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:18.998051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:18.998053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:18.998165 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:18.998578 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:18.998581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:18.998584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:18.998841 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:18.998968 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:18.999601 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 20:14:18.999778 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":460,"threads_started":1}
I20260620 20:14:19.001035 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:18.983324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34473 (local address 127.4.169.131:43551)
0620 20:14:18.984029 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:18.984077 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:18.985788 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:18.986545 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:18.986553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:18.987084 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:18.987650 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.987667 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.988771 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.988776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.989990 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.990002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.990926 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.990934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.991172 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.992006 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:18.992042 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:18.994528 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:18.997525 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:18.997532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:18.997543 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:18.997939 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:18.998253 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:18.998256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:18.998258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:18.998495 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:18.998949 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:18.998955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:18.999272 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 20:14:18.999493 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 20:14:18.999888 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":294,"threads_started":1}
I20260620 20:14:19.002619 22382 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34473
I20260620 20:14:19.002897 22382 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:19.003611 22382 heartbeater.cc:511] Master 127.4.169.190:34473 requested a full tablet report, sending...
I20260620 20:14:19.005163 21924 ts_manager.cc:194] Registered new tserver with Master: 6adf945b7a6448aa9d3212df10f5161f (127.4.169.131:43359)
I20260620 20:14:19.006364 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:14:19.006392 21924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:43551
I20260620 20:14:19.025146 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.011194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.4.169.190:34473)
0620 20:14:19.011399 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.011407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.011822 (+ 415us) server_negotiation.cc:407] Connection header received
0620 20:14:19.011951 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.011955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.012017 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.012107 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:19.013523 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.014293 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.015956 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.016388 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.016539 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.019267 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:19.019273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:19.019276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:19.019312 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:19.022273 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:19.023122 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:19.023126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:19.023129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:19.023264 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:19.023669 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:19.023673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:19.023675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:19.024041 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:19.024228 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.024570 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.024754 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:14:19.029934 21924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:19.038647 21924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:19.063354 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.051029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.4.169.129:39505)
0620 20:14:19.054377 (+ 3348us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.054422 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.054451 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:14:19.054538 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.054543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.054725 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.054921 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:19.055978 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.057689 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.060401 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.061919 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.062068 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.062315 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.062846 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.063012 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2991,"thread_start_us":2851,"threads_started":1}
I20260620 20:14:19.063529 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.053250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.4.169.131:43359)
0620 20:14:19.054352 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.054359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.054449 (+ 90us) server_negotiation.cc:407] Connection header received
0620 20:14:19.054549 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.054555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.054721 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.055791 (+ 1070us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:19.056088 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.057693 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.060963 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.062414 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.062543 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.062767 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.063017 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.063187 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":856,"threads_started":1}
I20260620 20:14:19.063591 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.052968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:43359 (local address 127.0.0.1:36586)
0620 20:14:19.053640 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.053661 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:19.053833 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:19.054975 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:19.054979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:19.055040 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:19.055583 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.055598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.058515 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.058523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.060758 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.060775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.062626 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.062633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.062821 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.062856 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:14:19.062963 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:14:19.063103 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":2,"spinlock_wait_cycles":3840,"thread_start_us":356,"threads_started":1}
I20260620 20:14:19.063674 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.050684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.0.0.1:35618)
0620 20:14:19.051437 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.051479 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:19.051639 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:19.054922 (+ 3283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:19.054929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:19.054958 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:19.055789 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.055797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.057815 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.057819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.060187 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.060203 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.062135 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.062143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.062460 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.062524 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 20:14:19.062663 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:14:19.063253 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":513,"spinlock_wait_cycles":14208,"thread_start_us":368,"threads_started":1}
I20260620 20:14:19.064163 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.050877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:37023 (local address 127.0.0.1:45038)
0620 20:14:19.052709 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.052723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:19.053088 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:19.054939 (+ 1851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:19.054947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:19.054975 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:19.055780 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.055794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.059114 (+ 3320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.059121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.060954 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.060968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.062350 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.062355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.062570 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.062594 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:14:19.062655 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 20:14:19.062905 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1741,"mutex_wait_us":126,"thread_start_us":1614,"threads_started":1}
I20260620 20:14:19.064864 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.051854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.4.169.130:37023)
0620 20:14:19.052475 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.052484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.052820 (+ 336us) server_negotiation.cc:407] Connection header received
0620 20:14:19.054450 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.054459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.054723 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.055322 (+ 599us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:19.057893 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.058975 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.061145 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.062125 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.064102 (+ 1977us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.064282 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.064348 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.064521 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":339,"threads_started":1}
I20260620 20:14:19.068210 22049 tablet_service.cc:1467] Processing CreateTablet for tablet 518a28d131a242aa9bf8e14aa09261c6 (DEFAULT_TABLE table=test-table [id=104cc6162fbd4bc5a993d677a86bdb08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:19.068276 22317 tablet_service.cc:1467] Processing CreateTablet for tablet 518a28d131a242aa9bf8e14aa09261c6 (DEFAULT_TABLE table=test-table [id=104cc6162fbd4bc5a993d677a86bdb08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:19.068331 22183 tablet_service.cc:1467] Processing CreateTablet for tablet 518a28d131a242aa9bf8e14aa09261c6 (DEFAULT_TABLE table=test-table [id=104cc6162fbd4bc5a993d677a86bdb08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:19.069703 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518a28d131a242aa9bf8e14aa09261c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:19.069701 22183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518a28d131a242aa9bf8e14aa09261c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:19.069707 22049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518a28d131a242aa9bf8e14aa09261c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:19.081694 22401 tablet_bootstrap.cc:492] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f: Bootstrap starting.
I20260620 20:14:19.085963 22401 tablet_bootstrap.cc:654] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:19.086071 22403 tablet_bootstrap.cc:492] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: Bootstrap starting.
I20260620 20:14:19.086886 22402 tablet_bootstrap.cc:492] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: Bootstrap starting.
I20260620 20:14:19.087636 22401 log.cc:824] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:19.090932 22402 tablet_bootstrap.cc:654] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:19.092197 22401 tablet_bootstrap.cc:492] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f: No bootstrap required, opened a new log
I20260620 20:14:19.092264 22403 tablet_bootstrap.cc:654] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:19.092348 22402 log.cc:824] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:19.092581 22401 ts_tablet_manager.cc:1397] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 20:14:19.093824 22403 log.cc:824] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:19.097878 22402 tablet_bootstrap.cc:492] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: No bootstrap required, opened a new log
I20260620 20:14:19.098264 22402 ts_tablet_manager.cc:1397] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260620 20:14:19.101116 22403 tablet_bootstrap.cc:492] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: No bootstrap required, opened a new log
I20260620 20:14:19.101435 22403 ts_tablet_manager.cc:1397] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260620 20:14:19.101282 22401 raft_consensus.cc:348] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } }
I20260620 20:14:19.101752 22401 raft_consensus.cc:374] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:19.101856 22401 raft_consensus.cc:729] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1aed67780f4abfa3384fa12159751f, State: Initialized, Role: FOLLOWER
I20260620 20:14:19.102581 22401 consensus_queue.cc:260] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } }
I20260620 20:14:19.104482 22401 ts_tablet_manager.cc:1428] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260620 20:14:19.109371 22402 raft_consensus.cc:348] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } }
I20260620 20:14:19.109897 22402 raft_consensus.cc:374] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:19.110001 22402 raft_consensus.cc:729] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44dfcf0aec86419fa4520b2d9e554a7f, State: Initialized, Role: FOLLOWER
I20260620 20:14:19.110724 22402 consensus_queue.cc:260] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } }
I20260620 20:14:19.111963 22248 heartbeater.cc:503] Master 127.4.169.190:34473 was elected leader, sending a full tablet report...
I20260620 20:14:19.112808 22402 ts_tablet_manager.cc:1428] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260620 20:14:19.112675 22403 raft_consensus.cc:348] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } }
I20260620 20:14:19.113380 22403 raft_consensus.cc:374] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:19.113535 22403 raft_consensus.cc:729] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adf945b7a6448aa9d3212df10f5161f, State: Initialized, Role: FOLLOWER
I20260620 20:14:19.114173 22403 consensus_queue.cc:260] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } }
I20260620 20:14:19.115309 22382 heartbeater.cc:503] Master 127.4.169.190:34473 was elected leader, sending a full tablet report...
I20260620 20:14:19.115813 22403 ts_tablet_manager.cc:1428] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
W20260620 20:14:19.172461 22115 tablet.cc:2369] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:19.198938 22249 tablet.cc:2369] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:19.209167 22408 raft_consensus.cc:482] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:19.209532 22408 raft_consensus.cc:504] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } }
I20260620 20:14:19.211333 22408 leader_election.cc:288] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea1aed67780f4abfa3384fa12159751f (127.4.169.129:39505), 6adf945b7a6448aa9d3212df10f5161f (127.4.169.131:43359)
I20260620 20:14:19.218222 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.211892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:60705 (local address 127.4.169.131:43359)
0620 20:14:19.212147 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.212156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.212843 (+ 687us) server_negotiation.cc:407] Connection header received
0620 20:14:19.213054 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.213060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.213137 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.213237 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:19.213777 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.214635 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.216490 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.217263 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.217449 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.217524 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.217818 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.217941 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 20:14:19.218370 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.211725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:37685 (local address 127.4.169.129:39505)
0620 20:14:19.211971 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.211980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.212713 (+ 733us) server_negotiation.cc:407] Connection header received
0620 20:14:19.213025 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.213028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.213134 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.213234 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:19.214084 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.214909 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.216670 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.217451 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.217626 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.217703 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.218001 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.218122 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:14:19.218472 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.211634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:43359 (local address 127.4.169.130:60705)
0620 20:14:19.212733 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.212747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:19.212885 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:19.213250 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:19.213255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:19.213287 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:19.213622 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.213632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.214756 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.214764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.216345 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.216357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.217475 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.217481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.217657 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.217680 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:14:19.217798 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:14:19.217932 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":43,"thread_start_us":582,"threads_started":1}
I20260620 20:14:19.218578 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.211567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.4.169.130:37685)
0620 20:14:19.212588 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.212608 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:19.212779 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:19.213578 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:19.213582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:19.213607 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:19.213938 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.213946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.215057 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.215064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.216513 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.216526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.217674 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.217680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.217843 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.217865 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:14:19.217971 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:14:19.218096 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":23,"thread_start_us":827,"threads_started":1}
I20260620 20:14:19.219545 22337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518a28d131a242aa9bf8e14aa09261c6" candidate_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adf945b7a6448aa9d3212df10f5161f" is_pre_election: true
I20260620 20:14:19.219594 22069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518a28d131a242aa9bf8e14aa09261c6" candidate_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1aed67780f4abfa3384fa12159751f" is_pre_election: true
I20260620 20:14:19.220088 22069 raft_consensus.cc:2454] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44dfcf0aec86419fa4520b2d9e554a7f in term 0.
I20260620 20:14:19.220078 22337 raft_consensus.cc:2454] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44dfcf0aec86419fa4520b2d9e554a7f in term 0.
I20260620 20:14:19.220747 22138 leader_election.cc:302] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44dfcf0aec86419fa4520b2d9e554a7f, ea1aed67780f4abfa3384fa12159751f; no voters:
I20260620 20:14:19.221175 22408 raft_consensus.cc:2781] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:19.221341 22408 raft_consensus.cc:482] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:19.221460 22408 raft_consensus.cc:3037] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:19.223956 22408 raft_consensus.cc:504] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } }
I20260620 20:14:19.224633 22408 leader_election.cc:288] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [CANDIDATE]: Term 1 election: Requested vote from peers ea1aed67780f4abfa3384fa12159751f (127.4.169.129:39505), 6adf945b7a6448aa9d3212df10f5161f (127.4.169.131:43359)
I20260620 20:14:19.225059 22069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518a28d131a242aa9bf8e14aa09261c6" candidate_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1aed67780f4abfa3384fa12159751f"
I20260620 20:14:19.225174 22337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518a28d131a242aa9bf8e14aa09261c6" candidate_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adf945b7a6448aa9d3212df10f5161f"
I20260620 20:14:19.225270 22069 raft_consensus.cc:3037] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:19.225425 22337 raft_consensus.cc:3037] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:19.227782 22069 raft_consensus.cc:2454] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44dfcf0aec86419fa4520b2d9e554a7f in term 1.
I20260620 20:14:19.227823 22337 raft_consensus.cc:2454] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44dfcf0aec86419fa4520b2d9e554a7f in term 1.
I20260620 20:14:19.228204 22138 leader_election.cc:302] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44dfcf0aec86419fa4520b2d9e554a7f, ea1aed67780f4abfa3384fa12159751f; no voters:
I20260620 20:14:19.228521 22408 raft_consensus.cc:2781] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:19.229005 22408 raft_consensus.cc:686] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 1 LEADER]: Becoming Leader. State: Replica: 44dfcf0aec86419fa4520b2d9e554a7f, State: Running, Role: LEADER
I20260620 20:14:19.229476 22408 consensus_queue.cc:237] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } }
W20260620 20:14:19.231429 22383 tablet.cc:2369] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:19.234133 21924 catalog_manager.cc:5495] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 44dfcf0aec86419fa4520b2d9e554a7f (127.4.169.130). New cstate: current_term: 1 leader_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44dfcf0aec86419fa4520b2d9e554a7f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 } health_report { overall_health: UNKNOWN } } }
I20260620 20:14:19.257270 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.251770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.4.169.130:37023)
0620 20:14:19.251993 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.251998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.252018 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:14:19.252182 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.252186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.252274 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.252425 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:19.252794 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.253541 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.255092 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.255564 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.255698 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.256172 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:19.256679 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:19.256842 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.256893 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.256973 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:14:19.265609 22337 raft_consensus.cc:1261] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [term 1 FOLLOWER]: Refusing update from remote peer 44dfcf0aec86419fa4520b2d9e554a7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:19.265610 22069 raft_consensus.cc:1261] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [term 1 FOLLOWER]: Refusing update from remote peer 44dfcf0aec86419fa4520b2d9e554a7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:19.266664 22412 consensus_queue.cc:1035] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1aed67780f4abfa3384fa12159751f" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:19.267061 22408 consensus_queue.cc:1035] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adf945b7a6448aa9d3212df10f5161f" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:19.278825 22415 mvcc.cc:205] Tried to move back new op lower bound from 7299016537141321728 to 7299016537007640576. Current Snapshot: MvccSnapshot[applied={T|T < 7299016537141321728}]
I20260620 20:14:19.281445 22417 mvcc.cc:205] Tried to move back new op lower bound from 7299016537141321728 to 7299016537007640576. Current Snapshot: MvccSnapshot[applied={T|T < 7299016537141321728}]
I20260620 20:14:19.282546 22416 mvcc.cc:205] Tried to move back new op lower bound from 7299016537141321728 to 7299016537007640576. Current Snapshot: MvccSnapshot[applied={T|T < 7299016537141321728}]
I20260620 20:14:19.297899 21924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:19.298197 21924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:19.302567 21924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3593bdaad5c04b04a77714615d102cbc: Sending DeleteTablet for 3 replicas of tablet 518a28d131a242aa9bf8e14aa09261c6
I20260620 20:14:19.303781 22183 tablet_service.cc:1514] Processing DeleteTablet for tablet 518a28d131a242aa9bf8e14aa09261c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:19 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:45038
I20260620 20:14:19.303778 22049 tablet_service.cc:1514] Processing DeleteTablet for tablet 518a28d131a242aa9bf8e14aa09261c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:19 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:35618
I20260620 20:14:19.303947 22317 tablet_service.cc:1514] Processing DeleteTablet for tablet 518a28d131a242aa9bf8e14aa09261c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:19 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:36586
I20260620 20:14:19.306602 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 21986
I20260620 20:14:19.333189 22427 tablet_replica.cc:330] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: stopping tablet replica
I20260620 20:14:19.334124 22427 raft_consensus.cc:2229] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:19.336022 22427 raft_consensus.cc:2258] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:19.335099 22426 tablet_replica.cc:330] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f: stopping tablet replica
I20260620 20:14:19.336869 22426 raft_consensus.cc:2229] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:19.337903 22426 raft_consensus.cc:2258] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:19.340590 22427 ts_tablet_manager.cc:1905] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:19.341670 22428 tablet_replica.cc:330] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: stopping tablet replica
Jun 20 20:14:19 dist-test-slave-4bwq krb5kdc[21880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:14:19.343791 22426 ts_tablet_manager.cc:1905] T 518a28d131a242aa9bf8e14aa09261c6 P ea1aed67780f4abfa3384fa12159751f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:19.344362 22428 raft_consensus.cc:2229] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:19.345876 22428 raft_consensus.cc:2258] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:19.349814 22427 ts_tablet_manager.cc:1918] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:19.350014 22427 log.cc:1196] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/518a28d131a242aa9bf8e14aa09261c6
I20260620 20:14:19.351119 22427 ts_tablet_manager.cc:1939] T 518a28d131a242aa9bf8e14aa09261c6 P 44dfcf0aec86419fa4520b2d9e554a7f: Deleting consensus metadata
I20260620 20:14:19.352118 22428 ts_tablet_manager.cc:1905] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:19.353016 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.307735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.4.169.129:39505)
0620 20:14:19.307929 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.307936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.332031 (+ 24095us) server_negotiation.cc:407] Connection header received
0620 20:14:19.332304 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.332311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.332424 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.332553 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:19.334554 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.335870 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.338082 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.338783 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.338890 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.344657 (+ 5767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:19.344674 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:19.344693 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:19.344746 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:19.349106 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:19.350176 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:19.350186 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:19.350190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:19.350414 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:19.350940 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:19.350945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:19.350949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:19.351429 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:19.351717 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.352336 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.352613 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:14:19.353215 21910 catalog_manager.cc:4841] TS 44dfcf0aec86419fa4520b2d9e554a7f (127.4.169.130:37023): tablet 518a28d131a242aa9bf8e14aa09261c6 (table test-table [id=104cc6162fbd4bc5a993d677a86bdb08]) successfully deleted
I20260620 20:14:19.354664 22109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:19.360908 22428 ts_tablet_manager.cc:1918] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:19.361234 22428 log.cc:1196] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/518a28d131a242aa9bf8e14aa09261c6
I20260620 20:14:19.361891 22428 ts_tablet_manager.cc:1939] T 518a28d131a242aa9bf8e14aa09261c6 P 6adf945b7a6448aa9d3212df10f5161f: Deleting consensus metadata
I20260620 20:14:19.364166 21910 catalog_manager.cc:4841] TS 6adf945b7a6448aa9d3212df10f5161f (127.4.169.131:43359): tablet 518a28d131a242aa9bf8e14aa09261c6 (table test-table [id=104cc6162fbd4bc5a993d677a86bdb08]) successfully deleted
I20260620 20:14:19.451766 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 21986
I20260620 20:14:19.470250 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 22120
W20260620 20:14:19.470610 21911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.169.129:39505 (error 108)
I20260620 20:14:19.473308 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.472900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.0.0.1:35630)
0620 20:14:19.473052 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.473174 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:39505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:14:19.473591 21911 catalog_manager.cc:4584] TS ea1aed67780f4abfa3384fa12159751f (127.4.169.129:39505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 518a28d131a242aa9bf8e14aa09261c6: Network error: Client connection negotiation failed: client connection to 127.4.169.129:39505: connect: Connection refused (error 111)
Jun 20 20:14:19 dist-test-slave-4bwq krb5kdc[21880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:14:19.491593 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.471634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.4.169.130:37023)
0620 20:14:19.471825 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.471831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.471851 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:14:19.471922 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.471929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.472049 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.472160 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:19.474089 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.475397 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.477438 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.478172 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.478322 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.483872 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:19.483888 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:19.483907 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:19.483965 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:19.488281 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:19.489214 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:19.489224 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:19.489228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:19.489414 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:19.489852 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:19.489857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:19.489861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:19.490326 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:19.490707 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.490960 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.491212 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:14:19.492358 22243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:19.527231 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.526739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.0.0.1:35642)
0620 20:14:19.526983 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.527078 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:39505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 20:14:19.528333 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.527954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.0.0.1:35656)
0620 20:14:19.528138 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.528211 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:39505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 20:14:19.589092 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.588609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.0.0.1:35662)
0620 20:14:19.588836 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.588937 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:39505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 20:14:19.590142 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.589833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.0.0.1:35678)
0620 20:14:19.589960 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.590031 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:39505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 20:14:19.610996 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 22120
I20260620 20:14:19.630293 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 22254
Jun 20 20:14:19 dist-test-slave-4bwq krb5kdc[21880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:14:19.645907 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.631238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.4.169.131:43359)
0620 20:14:19.631435 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.631441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.631461 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:14:19.631678 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.631685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.631761 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.631868 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:19.633162 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.633884 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.635278 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.635676 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.635808 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.640386 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:19.640401 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:19.640416 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:19.640448 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:19.643179 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:19.643856 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:19.643863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:19.643865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:19.644009 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:19.644384 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:19.644390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:19.644393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:19.644730 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:19.645028 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:19.645338 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 20:14:19.645564 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:14:19.646660 22377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:19.670511 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.669946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.0.0.1:35684)
0620 20:14:19.670173 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.670271 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:39505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 20:14:19.671579 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.671189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.0.0.1:35694)
0620 20:14:19.671372 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.671444 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:39505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 20:14:19.736753 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 22254
I20260620 20:14:19.753531 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 21893
I20260620 20:14:19.754662 21955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:19.866745 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 21893
I20260620 20:14:19.866773 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.866269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:39505 (local address 127.0.0.1:35710)
0620 20:14:19.866464 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.866569 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:39505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
2026-06-20T20:14:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4391 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T20:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:19Z Disabled control of system clock
I20260620 20:14:19.945915 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42765
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:42765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:14:20.281203 22438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:20.281497 22438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:20.281580 22438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:20.288810 22438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:20.288918 22438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:20.288957 22438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:20.289014 22438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:20.289052 22438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:20.298024 22438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:42765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:20.299309 22438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:20.301014 22438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:20.312693 22443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:20.313712 22438 server_base.cc:1034] running on GCE node
W20260620 20:14:20.313755 22444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:20.313779 22446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:20.314723 22438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:20.316207 22438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:20.317533 22438 hybrid_clock.cc:648] HybridClock initialized: now 1781986460317478 us; error 45 us; skew 500 ppm
I20260620 20:14:20.317993 22438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:20.325095 22438 webserver.cc:460] Webserver started at http://127.4.169.190:34261/ using document root <none> and password file <none>
I20260620 20:14:20.325867 22438 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:20.326016 22438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:20.326350 22438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:20.329401 22438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "b5e0eae39ef64d35965404a10a941cc7"
format_stamp: "Formatted at 2026-06-20 20:14:20 on dist-test-slave-4bwq"
server_key: "f8ae89cf81b007032b6c1bdb205f37eb"
server_key_iv: "57dc1f7383c9cc82d437616db074f8dc"
server_key_version: "encryptionkey@0"
I20260620 20:14:20.330200 22438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "b5e0eae39ef64d35965404a10a941cc7"
format_stamp: "Formatted at 2026-06-20 20:14:20 on dist-test-slave-4bwq"
server_key: "f8ae89cf81b007032b6c1bdb205f37eb"
server_key_iv: "57dc1f7383c9cc82d437616db074f8dc"
server_key_version: "encryptionkey@0"
I20260620 20:14:20.335340 22438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:14:20.338886 22452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:20.340188 22438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:14:20.340374 22438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "b5e0eae39ef64d35965404a10a941cc7"
format_stamp: "Formatted at 2026-06-20 20:14:20 on dist-test-slave-4bwq"
server_key: "f8ae89cf81b007032b6c1bdb205f37eb"
server_key_iv: "57dc1f7383c9cc82d437616db074f8dc"
server_key_version: "encryptionkey@0"
I20260620 20:14:20.340593 22438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:20.369951 22438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:20.370998 22438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:20.371290 22438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:20.389765 22438 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:42765
I20260620 20:14:20.389786 22503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:42765 every 8 connection(s)
I20260620 20:14:20.391155 22438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:20.394871 22504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:20.399371 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 22438
I20260620 20:14:20.399574 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:20.400022 4774 external_mini_cluster.cc:1428] Setting key d284a3e5ab9a2d29014631f10a751dc1
I20260620 20:14:20.407748 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7: Bootstrap starting.
I20260620 20:14:20.411599 22504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:20.412998 22504 log.cc:824] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:20.416574 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7: No bootstrap required, opened a new log
I20260620 20:14:20.416796 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:20.402120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.4.169.190:42765)
0620 20:14:20.403373 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:20.403393 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:20.403446 (+ 53us) server_negotiation.cc:407] Connection header received
0620 20:14:20.406011 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:20.406041 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:20.406688 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:20.407160 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:20.407737 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.409742 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.411366 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.412508 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.412716 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:20.413920 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:20.413933 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:20.413959 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:20.414691 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:20.414796 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:20.414827 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:20.414939 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:20.415164 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 20:14:20.415498 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":477,"threads_started":1}
I20260620 20:14:20.424625 22504 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e0eae39ef64d35965404a10a941cc7" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42765 } }
I20260620 20:14:20.424993 22504 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:20.425069 22504 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e0eae39ef64d35965404a10a941cc7, State: Initialized, Role: FOLLOWER
I20260620 20:14:20.425756 22504 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e0eae39ef64d35965404a10a941cc7" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42765 } }
I20260620 20:14:20.426038 22504 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:20.426210 22504 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:20.426505 22504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:20.428714 22504 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e0eae39ef64d35965404a10a941cc7" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42765 } }
I20260620 20:14:20.429289 22504 leader_election.cc:302] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5e0eae39ef64d35965404a10a941cc7; no voters:
I20260620 20:14:20.429878 22504 leader_election.cc:288] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:20.430040 22509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:20.430936 22509 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [term 1 LEADER]: Becoming Leader. State: Replica: b5e0eae39ef64d35965404a10a941cc7, State: Running, Role: LEADER
I20260620 20:14:20.431543 22509 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e0eae39ef64d35965404a10a941cc7" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42765 } }
I20260620 20:14:20.432240 22504 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:20.434763 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5e0eae39ef64d35965404a10a941cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e0eae39ef64d35965404a10a941cc7" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42765 } } }
I20260620 20:14:20.435266 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:20.435788 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5e0eae39ef64d35965404a10a941cc7. Latest consensus state: current_term: 1 leader_uuid: "b5e0eae39ef64d35965404a10a941cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e0eae39ef64d35965404a10a941cc7" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42765 } } }
I20260620 20:14:20.436303 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:20.438174 22517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:20.442479 22517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:20.451442 22517 catalog_manager.cc:1347] Generated new cluster ID: 36e8f6da60394676ae4225cc68a63923
I20260620 20:14:20.451561 22517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:20.463131 22517 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:20.464479 22517 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:20.481310 22517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7: Generated new TSK 0
I20260620 20:14:20.482236 22517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:20.493870 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42765
--builtin_ntp_servers=127.4.169.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:14:20.836066 22528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:20.836449 22528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:20.836609 22528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:20.844177 22528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:20.844273 22528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:20.844376 22528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:14:20.853533 22528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:20.854809 22528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:20.856563 22528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:20.869652 22533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:20.869714 22536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:20.869689 22534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:20.870972 22528 server_base.cc:1034] running on GCE node
I20260620 20:14:20.871556 22528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:20.872596 22528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:20.873850 22528 hybrid_clock.cc:648] HybridClock initialized: now 1781986460873824 us; error 50 us; skew 500 ppm
I20260620 20:14:20.874225 22528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:20.876577 22528 webserver.cc:460] Webserver started at http://127.4.169.129:41625/ using document root <none> and password file <none>
I20260620 20:14:20.877362 22528 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:20.877491 22528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:20.877806 22528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:20.880693 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "76ea42dbdb2f4f379dba75bf8d0e2368"
format_stamp: "Formatted at 2026-06-20 20:14:20 on dist-test-slave-4bwq"
server_key: "818ffa3d48c81758f806152c7aba2d47"
server_key_iv: "3ce6144f2a4b82383c8c6a9dcfcffd52"
server_key_version: "encryptionkey@0"
I20260620 20:14:20.881479 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "76ea42dbdb2f4f379dba75bf8d0e2368"
format_stamp: "Formatted at 2026-06-20 20:14:20 on dist-test-slave-4bwq"
server_key: "818ffa3d48c81758f806152c7aba2d47"
server_key_iv: "3ce6144f2a4b82383c8c6a9dcfcffd52"
server_key_version: "encryptionkey@0"
I20260620 20:14:20.886684 22528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:14:20.890244 22542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:20.891599 22528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:20.891777 22528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "76ea42dbdb2f4f379dba75bf8d0e2368"
format_stamp: "Formatted at 2026-06-20 20:14:20 on dist-test-slave-4bwq"
server_key: "818ffa3d48c81758f806152c7aba2d47"
server_key_iv: "3ce6144f2a4b82383c8c6a9dcfcffd52"
server_key_version: "encryptionkey@0"
I20260620 20:14:20.892004 22528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:20.914529 22528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:20.915393 22528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:20.915719 22528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:20.916893 22528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:20.918974 22528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:20.919068 22528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:20.919149 22528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:20.919206 22528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:20.953594 22528 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:32957
I20260620 20:14:20.953715 22654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:32957 every 8 connection(s)
I20260620 20:14:20.955031 22528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:20.958838 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 22528
I20260620 20:14:20.959043 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:20.959623 4774 external_mini_cluster.cc:1428] Setting key aba5d01762e23d72d22c3f065090076d
I20260620 20:14:20.962842 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42765
--builtin_ntp_servers=127.4.169.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:14:20.969827 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:20.957781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:38895 (local address 127.4.169.190:42765)
0620 20:14:20.958270 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:20.958274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:20.959796 (+ 1522us) server_negotiation.cc:407] Connection header received
0620 20:14:20.961868 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:20.961872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:20.961939 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:20.962027 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:20.963561 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.964258 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.966754 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.967191 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.967300 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:20.969014 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:20.969020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:20.969025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:20.969049 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:20.969093 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:20.969106 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:20.969175 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:20.969451 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 20:14:20.969543 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":268,"threads_started":1}
I20260620 20:14:20.971377 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:20.958473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42765 (local address 127.4.169.129:38895)
0620 20:14:20.959608 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:20.959659 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:20.961560 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:20.962307 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:20.962321 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:20.962485 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:20.963342 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:20.963369 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.964422 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.964430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:20.966598 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:20.966616 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.967445 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.967456 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:20.967795 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:20.968609 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:20.968614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:20.968680 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:20.968796 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:20.969206 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:20.969221 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:20.969233 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 20:14:20.969424 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 20:14:20.969899 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":294,"threads_started":1}
I20260620 20:14:20.973197 22655 heartbeater.cc:347] Connected to a master server at 127.4.169.190:42765
I20260620 20:14:20.973520 22655 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:20.974310 22655 heartbeater.cc:511] Master 127.4.169.190:42765 requested a full tablet report, sending...
I20260620 20:14:20.976711 22469 ts_manager.cc:194] Registered new tserver with Master: 76ea42dbdb2f4f379dba75bf8d0e2368 (127.4.169.129:32957)
I20260620 20:14:20.979054 22469 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.129:38895
W20260620 20:14:21.304769 22659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:21.305065 22659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:21.305155 22659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:21.312495 22659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:21.312603 22659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:21.312779 22659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:14:21.322045 22659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:21.323303 22659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:21.324993 22659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:21.338071 22664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:21.338030 22667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:21.338106 22665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:21.339485 22659 server_base.cc:1034] running on GCE node
I20260620 20:14:21.340193 22659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:21.341282 22659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:21.342494 22659 hybrid_clock.cc:648] HybridClock initialized: now 1781986461342449 us; error 76 us; skew 500 ppm
I20260620 20:14:21.342818 22659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:21.345126 22659 webserver.cc:460] Webserver started at http://127.4.169.130:38263/ using document root <none> and password file <none>
I20260620 20:14:21.345819 22659 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:21.345952 22659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:21.346273 22659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:21.349092 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "07fc776b1ebf4d8b80487a723c651e55"
format_stamp: "Formatted at 2026-06-20 20:14:21 on dist-test-slave-4bwq"
server_key: "462aa79f50133178da5367bb07f6264f"
server_key_iv: "de315c57c7c9f464637a006032d84a21"
server_key_version: "encryptionkey@0"
I20260620 20:14:21.349861 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "07fc776b1ebf4d8b80487a723c651e55"
format_stamp: "Formatted at 2026-06-20 20:14:21 on dist-test-slave-4bwq"
server_key: "462aa79f50133178da5367bb07f6264f"
server_key_iv: "de315c57c7c9f464637a006032d84a21"
server_key_version: "encryptionkey@0"
I20260620 20:14:21.354959 22659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:14:21.358496 22673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:21.359845 22659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:21.360023 22659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "07fc776b1ebf4d8b80487a723c651e55"
format_stamp: "Formatted at 2026-06-20 20:14:21 on dist-test-slave-4bwq"
server_key: "462aa79f50133178da5367bb07f6264f"
server_key_iv: "de315c57c7c9f464637a006032d84a21"
server_key_version: "encryptionkey@0"
I20260620 20:14:21.360251 22659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:21.374315 22659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:21.375195 22659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:21.375509 22659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:21.376443 22659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:21.377864 22659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:21.377985 22659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:21.378126 22659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:21.378216 22659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:21.415113 22659 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:45263
I20260620 20:14:21.415162 22785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:45263 every 8 connection(s)
I20260620 20:14:21.416456 22659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:21.422437 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 22659
I20260620 20:14:21.422752 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:21.423211 4774 external_mini_cluster.cc:1428] Setting key 6c008db57a391b52f0794d912ddc0c65
I20260620 20:14:21.426535 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42765
--builtin_ntp_servers=127.4.169.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:14:21.430634 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.419094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:55731 (local address 127.4.169.190:42765)
0620 20:14:21.419274 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:21.419279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:21.420641 (+ 1362us) server_negotiation.cc:407] Connection header received
0620 20:14:21.422577 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:21.422583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:21.422700 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:21.422825 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:21.424101 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.425166 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.427709 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.428108 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.428216 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.429852 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:21.429857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:21.429860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:21.429880 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:21.429913 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:21.429926 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:21.430006 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:21.430263 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 20:14:21.430346 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:14:21.432155 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.419479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42765 (local address 127.4.169.130:55731)
0620 20:14:21.420452 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:21.420500 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:21.422242 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:21.423071 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:21.423080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:21.423172 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:21.423897 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.423923 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.425340 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.425345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.427535 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.427555 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.428348 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.428358 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.428694 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.429496 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:21.429499 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:21.429556 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:21.429659 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:21.430038 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:21.430051 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:21.430063 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 20:14:21.430247 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 20:14:21.430712 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":363,"threads_started":1}
I20260620 20:14:21.433764 22786 heartbeater.cc:347] Connected to a master server at 127.4.169.190:42765
I20260620 20:14:21.434095 22786 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:21.434939 22786 heartbeater.cc:511] Master 127.4.169.190:42765 requested a full tablet report, sending...
I20260620 20:14:21.436414 22469 ts_manager.cc:194] Registered new tserver with Master: 07fc776b1ebf4d8b80487a723c651e55 (127.4.169.130:45263)
I20260620 20:14:21.437703 22469 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.130:55731
W20260620 20:14:21.758548 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:21.758816 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:21.758903 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:21.766343 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:21.766461 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:21.766569 22789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:14:21.775667 22789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:42765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:21.776855 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:21.778620 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:21.791414 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:21.791448 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:21.791378 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:21.792461 22789 server_base.cc:1034] running on GCE node
I20260620 20:14:21.793015 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:21.793874 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:21.795091 22789 hybrid_clock.cc:648] HybridClock initialized: now 1781986461795041 us; error 37 us; skew 500 ppm
I20260620 20:14:21.795439 22789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:21.797719 22789 webserver.cc:460] Webserver started at http://127.4.169.131:41071/ using document root <none> and password file <none>
I20260620 20:14:21.798353 22789 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:21.798523 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:21.798799 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:21.801545 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "2e2a693d42ff42d0bb725954ed347613"
format_stamp: "Formatted at 2026-06-20 20:14:21 on dist-test-slave-4bwq"
server_key: "d86b7b42d084be585e1a71e6e1d70b86"
server_key_iv: "b645d6b12a36cdb885d16d0c60e03dae"
server_key_version: "encryptionkey@0"
I20260620 20:14:21.802351 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "2e2a693d42ff42d0bb725954ed347613"
format_stamp: "Formatted at 2026-06-20 20:14:21 on dist-test-slave-4bwq"
server_key: "d86b7b42d084be585e1a71e6e1d70b86"
server_key_iv: "b645d6b12a36cdb885d16d0c60e03dae"
server_key_version: "encryptionkey@0"
I20260620 20:14:21.807341 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 20:14:21.810696 22803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:21.812044 22789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:14:21.812220 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "2e2a693d42ff42d0bb725954ed347613"
format_stamp: "Formatted at 2026-06-20 20:14:21 on dist-test-slave-4bwq"
server_key: "d86b7b42d084be585e1a71e6e1d70b86"
server_key_iv: "b645d6b12a36cdb885d16d0c60e03dae"
server_key_version: "encryptionkey@0"
I20260620 20:14:21.812440 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:21.827416 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:21.828266 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:21.828586 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:21.829543 22789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:21.831063 22789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:21.831182 22789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:21.831317 22789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:21.831405 22789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:21.865063 22789 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:45163
I20260620 20:14:21.865105 22915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:45163 every 8 connection(s)
I20260620 20:14:21.866343 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:21.873382 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 22789
I20260620 20:14:21.873569 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:21.874054 4774 external_mini_cluster.cc:1428] Setting key f2415168faae947274305bcccbfd21ac
I20260620 20:14:21.879361 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.868952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:57735 (local address 127.4.169.190:42765)
0620 20:14:21.869162 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:21.869168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:21.870623 (+ 1455us) server_negotiation.cc:407] Connection header received
0620 20:14:21.872339 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:21.872344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:21.872409 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:21.872497 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:21.873766 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.874714 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.876925 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.877370 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.877475 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.878712 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:21.878717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:21.878720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:21.878737 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:21.878768 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:21.878779 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:21.878867 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:21.879056 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 20:14:21.879128 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:14:21.880273 22918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.869516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42765 (local address 127.4.169.131:57735)
0620 20:14:21.870381 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:21.870514 (+ 133us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:21.872099 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:21.872692 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:21.872699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:21.872782 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:21.873589 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.873616 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.874923 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.874927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.876773 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.876788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.877564 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.877570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.877759 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.878358 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:21.878360 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:21.878448 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:21.878543 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:21.878876 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:21.878883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:21.878889 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:14:21.879025 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 20:14:21.879287 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":264,"threads_started":1}
I20260620 20:14:21.881584 22916 heartbeater.cc:347] Connected to a master server at 127.4.169.190:42765
I20260620 20:14:21.881870 22916 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:21.882651 22916 heartbeater.cc:511] Master 127.4.169.190:42765 requested a full tablet report, sending...
I20260620 20:14:21.884001 22469 ts_manager.cc:194] Registered new tserver with Master: 2e2a693d42ff42d0bb725954ed347613 (127.4.169.131:45163)
I20260620 20:14:21.884799 22469 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.131:57735
I20260620 20:14:21.889210 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:14:21.898836 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.893136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.4.169.190:42765)
0620 20:14:21.893284 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:21.893288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:21.893794 (+ 506us) server_negotiation.cc:407] Connection header received
0620 20:14:21.893922 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:21.893926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:21.893992 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:21.894074 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:21.894601 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.895272 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.896709 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.897102 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.897202 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.898184 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:21.898189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:21.898192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:21.898209 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:21.898237 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:21.898248 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:21.898317 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:21.898534 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 20:14:21.898602 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:14:21.902486 22469 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:21.909358 22469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:21.936776 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.921143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:45163 (local address 127.0.0.1:60878)
0620 20:14:21.923365 (+ 2222us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:21.923426 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:21.923649 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:21.925933 (+ 2284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:21.925945 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:21.925988 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:21.926480 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.926507 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.929154 (+ 2647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.929163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.932482 (+ 3319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.932512 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.935520 (+ 3008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.935529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.935830 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.935902 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 20:14:21.936041 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:14:21.936304 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1906,"thread_start_us":15256,"threads_started":1}
I20260620 20:14:21.937716 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.921264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.4.169.131:45163)
0620 20:14:21.924787 (+ 3523us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:21.924799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:21.924843 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:14:21.924964 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:21.924975 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:21.925323 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:21.925562 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:21.926778 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.928956 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.932779 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.934799 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.934965 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.935342 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:21.936709 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0620 20:14:21.936944 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3304,"thread_start_us":2761,"threads_started":1}
I20260620 20:14:21.941953 22851 tablet_service.cc:1467] Processing CreateTablet for tablet c94dc08d8fc94d5893b6eb69cb2a60d2 (DEFAULT_TABLE table=test-table [id=ffc33b70d122442f9dbbadc30b1296a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:21.943792 22851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94dc08d8fc94d5893b6eb69cb2a60d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:21.944370 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.921335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.0.0.1:53198)
0620 20:14:21.934433 (+ 13098us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:21.934459 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:21.935971 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:21.937134 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:21.937141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:21.937169 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:21.937755 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.937770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.939813 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.939820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.941444 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.941457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.943404 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.943411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.943596 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.943630 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:14:21.943731 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:14:21.943878 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12887,"mutex_wait_us":108,"thread_start_us":2231,"threads_started":1}
I20260620 20:14:21.944373 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.921881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.4.169.129:32957)
0620 20:14:21.922672 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:21.922685 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:21.934610 (+ 11925us) server_negotiation.cc:407] Connection header received
0620 20:14:21.936196 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:21.936206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:21.936521 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:21.936701 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:21.937966 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.939624 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.941647 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.942894 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.943041 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.943269 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:21.943801 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 20:14:21.943977 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":370,"threads_started":1}
I20260620 20:14:21.945065 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.923327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:45263 (local address 127.0.0.1:33464)
0620 20:14:21.936702 (+ 13375us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:21.936720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:21.936837 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:21.937455 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:21.937462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:21.937486 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:21.937994 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.938008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.939832 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.939839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.942106 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.942120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.944255 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.944262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.944479 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.944538 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:14:21.944616 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 20:14:21.944741 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13237,"mutex_wait_us":18,"thread_start_us":11318,"threads_started":1}
I20260620 20:14:21.945456 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.923668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.4.169.130:45263)
0620 20:14:21.936094 (+ 12426us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:21.936105 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:21.936962 (+ 857us) server_negotiation.cc:407] Connection header received
0620 20:14:21.937078 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:21.937085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:21.937275 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:21.937470 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:21.938175 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.939692 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.942304 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.943781 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.943895 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.944121 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:21.944905 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 20:14:21.945148 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12233,"thread_start_us":12430,"threads_started":1}
I20260620 20:14:21.948639 22721 tablet_service.cc:1467] Processing CreateTablet for tablet c94dc08d8fc94d5893b6eb69cb2a60d2 (DEFAULT_TABLE table=test-table [id=ffc33b70d122442f9dbbadc30b1296a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:21.948575 22590 tablet_service.cc:1467] Processing CreateTablet for tablet c94dc08d8fc94d5893b6eb69cb2a60d2 (DEFAULT_TABLE table=test-table [id=ffc33b70d122442f9dbbadc30b1296a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:21.949791 22721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94dc08d8fc94d5893b6eb69cb2a60d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:21.949891 22590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c94dc08d8fc94d5893b6eb69cb2a60d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:21.964572 22934 tablet_bootstrap.cc:492] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: Bootstrap starting.
I20260620 20:14:21.967918 22935 tablet_bootstrap.cc:492] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: Bootstrap starting.
I20260620 20:14:21.971009 22935 tablet_bootstrap.cc:654] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:21.971279 22934 tablet_bootstrap.cc:654] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:21.972265 22936 tablet_bootstrap.cc:492] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: Bootstrap starting.
I20260620 20:14:21.972368 22935 log.cc:824] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:21.972708 22934 log.cc:824] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:21.975751 22936 tablet_bootstrap.cc:654] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:21.975920 22935 tablet_bootstrap.cc:492] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: No bootstrap required, opened a new log
I20260620 20:14:21.976256 22935 ts_tablet_manager.cc:1397] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 20:14:21.976953 22936 log.cc:824] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:21.980037 22936 tablet_bootstrap.cc:492] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: No bootstrap required, opened a new log
I20260620 20:14:21.980150 22934 tablet_bootstrap.cc:492] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: No bootstrap required, opened a new log
I20260620 20:14:21.980402 22934 ts_tablet_manager.cc:1397] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260620 20:14:21.980400 22936 ts_tablet_manager.cc:1397] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260620 20:14:21.982435 22655 heartbeater.cc:503] Master 127.4.169.190:42765 was elected leader, sending a full tablet report...
I20260620 20:14:21.985603 22935 raft_consensus.cc:348] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } }
I20260620 20:14:21.986199 22935 raft_consensus.cc:374] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:21.986357 22935 raft_consensus.cc:729] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fc776b1ebf4d8b80487a723c651e55, State: Initialized, Role: FOLLOWER
I20260620 20:14:21.987241 22935 consensus_queue.cc:260] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } }
I20260620 20:14:21.991293 22934 raft_consensus.cc:348] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } }
I20260620 20:14:21.991427 22936 raft_consensus.cc:348] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } }
I20260620 20:14:21.991796 22936 raft_consensus.cc:374] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:21.991827 22934 raft_consensus.cc:374] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:21.991947 22934 raft_consensus.cc:729] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2a693d42ff42d0bb725954ed347613, State: Initialized, Role: FOLLOWER
I20260620 20:14:21.991964 22936 raft_consensus.cc:729] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ea42dbdb2f4f379dba75bf8d0e2368, State: Initialized, Role: FOLLOWER
I20260620 20:14:21.992075 22786 heartbeater.cc:503] Master 127.4.169.190:42765 was elected leader, sending a full tablet report...
I20260620 20:14:21.992630 22934 consensus_queue.cc:260] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } }
I20260620 20:14:21.992842 22935 ts_tablet_manager.cc:1428] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260620 20:14:21.992745 22936 consensus_queue.cc:260] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } }
I20260620 20:14:21.994278 22916 heartbeater.cc:503] Master 127.4.169.190:42765 was elected leader, sending a full tablet report...
I20260620 20:14:21.994609 22936 ts_tablet_manager.cc:1428] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260620 20:14:21.994810 22934 ts_tablet_manager.cc:1428] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260620 20:14:22.118111 22917 tablet.cc:2369] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:22.168519 22787 tablet.cc:2369] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:22.207634 22656 tablet.cc:2369] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:22.236763 22941 raft_consensus.cc:482] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:22.237133 22941 raft_consensus.cc:504] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } }
I20260620 20:14:22.238982 22941 leader_election.cc:288] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76ea42dbdb2f4f379dba75bf8d0e2368 (127.4.169.129:32957), 07fc776b1ebf4d8b80487a723c651e55 (127.4.169.130:45263)
I20260620 20:14:22.246362 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.239611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:36545 (local address 127.4.169.130:45263)
0620 20:14:22.239844 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.239852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.239880 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:14:22.239959 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.239964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.240058 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.240176 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:22.240707 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.241711 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.244189 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.245435 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.245611 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.245703 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.245950 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.246087 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:14:22.246475 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.239361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:59833 (local address 127.4.169.129:32957)
0620 20:14:22.239598 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.239606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.240313 (+ 707us) server_negotiation.cc:407] Connection header received
0620 20:14:22.240431 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.240435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.240503 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.240575 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:22.241089 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.242013 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.244385 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.245611 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.245752 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.245803 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.246068 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.246190 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 20:14:22.246454 22918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.239295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:45263 (local address 127.4.169.131:36545)
0620 20:14:22.239523 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.239544 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:22.239666 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:22.240242 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:22.240250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:22.240283 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:22.240523 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.240530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.241843 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.241847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.243997 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.244012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.245589 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.245593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.245778 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.245813 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:14:22.245906 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 20:14:22.246056 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":2,"spinlock_wait_cycles":21376}
I20260620 20:14:22.246457 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.239420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.4.169.131:59833)
0620 20:14:22.240193 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.240208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:22.240316 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:22.240621 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:22.240627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:22.240656 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:22.240929 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.240936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.242149 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.242153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.244222 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.244232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.245758 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.245764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.245917 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.245946 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:14:22.246029 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:14:22.246128 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":435,"threads_started":1}
I20260620 20:14:22.247705 22610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94dc08d8fc94d5893b6eb69cb2a60d2" candidate_uuid: "2e2a693d42ff42d0bb725954ed347613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" is_pre_election: true
I20260620 20:14:22.247677 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94dc08d8fc94d5893b6eb69cb2a60d2" candidate_uuid: "2e2a693d42ff42d0bb725954ed347613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fc776b1ebf4d8b80487a723c651e55" is_pre_election: true
I20260620 20:14:22.248296 22610 raft_consensus.cc:2454] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e2a693d42ff42d0bb725954ed347613 in term 0.
I20260620 20:14:22.248296 22741 raft_consensus.cc:2454] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e2a693d42ff42d0bb725954ed347613 in term 0.
I20260620 20:14:22.249018 22805 leader_election.cc:302] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07fc776b1ebf4d8b80487a723c651e55, 2e2a693d42ff42d0bb725954ed347613, 76ea42dbdb2f4f379dba75bf8d0e2368; no voters:
I20260620 20:14:22.249455 22941 raft_consensus.cc:2781] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:22.249603 22941 raft_consensus.cc:482] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:22.249722 22941 raft_consensus.cc:3037] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:22.251962 22941 raft_consensus.cc:504] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } }
I20260620 20:14:22.252565 22941 leader_election.cc:288] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [CANDIDATE]: Term 1 election: Requested vote from peers 76ea42dbdb2f4f379dba75bf8d0e2368 (127.4.169.129:32957), 07fc776b1ebf4d8b80487a723c651e55 (127.4.169.130:45263)
I20260620 20:14:22.253023 22610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94dc08d8fc94d5893b6eb69cb2a60d2" candidate_uuid: "2e2a693d42ff42d0bb725954ed347613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368"
I20260620 20:14:22.253054 22741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c94dc08d8fc94d5893b6eb69cb2a60d2" candidate_uuid: "2e2a693d42ff42d0bb725954ed347613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fc776b1ebf4d8b80487a723c651e55"
I20260620 20:14:22.253240 22741 raft_consensus.cc:3037] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:22.253240 22610 raft_consensus.cc:3037] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:22.256476 22741 raft_consensus.cc:2454] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2a693d42ff42d0bb725954ed347613 in term 1.
I20260620 20:14:22.256476 22610 raft_consensus.cc:2454] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2a693d42ff42d0bb725954ed347613 in term 1.
I20260620 20:14:22.256917 22805 leader_election.cc:302] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07fc776b1ebf4d8b80487a723c651e55, 2e2a693d42ff42d0bb725954ed347613, 76ea42dbdb2f4f379dba75bf8d0e2368; no voters:
I20260620 20:14:22.257320 22941 raft_consensus.cc:2781] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:22.257786 22941 raft_consensus.cc:686] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 1 LEADER]: Becoming Leader. State: Replica: 2e2a693d42ff42d0bb725954ed347613, State: Running, Role: LEADER
I20260620 20:14:22.258260 22941 consensus_queue.cc:237] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } }
I20260620 20:14:22.262938 22468 catalog_manager.cc:5495] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e2a693d42ff42d0bb725954ed347613 (127.4.169.131). New cstate: current_term: 1 leader_uuid: "2e2a693d42ff42d0bb725954ed347613" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e2a693d42ff42d0bb725954ed347613" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 } health_report { overall_health: UNKNOWN } } }
I20260620 20:14:22.317473 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.312375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60880 (local address 127.4.169.131:45163)
0620 20:14:22.312541 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.312545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.312563 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:14:22.312697 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.312701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.312777 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.312900 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:22.313236 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.313958 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.315355 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.315758 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.315847 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.316353 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:22.316868 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:22.316986 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.317113 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.317180 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:14:22.325392 22741 raft_consensus.cc:1261] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [term 1 FOLLOWER]: Refusing update from remote peer 2e2a693d42ff42d0bb725954ed347613: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:22.325392 22610 raft_consensus.cc:1261] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [term 1 FOLLOWER]: Refusing update from remote peer 2e2a693d42ff42d0bb725954ed347613: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:22.326447 22944 consensus_queue.cc:1035] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ea42dbdb2f4f379dba75bf8d0e2368" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 32957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:22.326802 22941 consensus_queue.cc:1035] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07fc776b1ebf4d8b80487a723c651e55" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:22.337682 22947 mvcc.cc:205] Tried to move back new op lower bound from 7299016549675474944 to 7299016549413519360. Current Snapshot: MvccSnapshot[applied={T|T < 7299016549675474944}]
I20260620 20:14:22.350602 22469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:22.350865 22469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:22.354285 22469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5e0eae39ef64d35965404a10a941cc7: Sending DeleteTablet for 3 replicas of tablet c94dc08d8fc94d5893b6eb69cb2a60d2
I20260620 20:14:22.355391 22851 tablet_service.cc:1514] Processing DeleteTablet for tablet c94dc08d8fc94d5893b6eb69cb2a60d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:22 UTC) from {username='slave'} at 127.0.0.1:60878
I20260620 20:14:22.355463 22590 tablet_service.cc:1514] Processing DeleteTablet for tablet c94dc08d8fc94d5893b6eb69cb2a60d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:22 UTC) from {username='slave'} at 127.0.0.1:53198
I20260620 20:14:22.355604 22721 tablet_service.cc:1514] Processing DeleteTablet for tablet c94dc08d8fc94d5893b6eb69cb2a60d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:22 UTC) from {username='slave'} at 127.0.0.1:33464
I20260620 20:14:22.357234 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 22528
I20260620 20:14:22.368192 22958 tablet_replica.cc:330] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: stopping tablet replica
I20260620 20:14:22.368357 22957 tablet_replica.cc:330] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: stopping tablet replica
I20260620 20:14:22.368736 22958 raft_consensus.cc:2229] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:22.368822 22957 raft_consensus.cc:2229] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:22.369407 22957 raft_consensus.cc:2258] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:22.370147 22958 raft_consensus.cc:2258] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:22.371343 22956 tablet_replica.cc:330] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: stopping tablet replica
I20260620 20:14:22.371795 22956 raft_consensus.cc:2229] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:22.372200 22957 ts_tablet_manager.cc:1905] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:22.372537 22956 raft_consensus.cc:2258] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:22.373498 22958 ts_tablet_manager.cc:1905] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:22.379679 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.359101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.4.169.129:32957)
0620 20:14:22.359326 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.359336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.370226 (+ 10890us) server_negotiation.cc:407] Connection header received
0620 20:14:22.370824 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.370830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.370921 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.371042 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:22.371726 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.372796 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.375120 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.375956 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.376099 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.377621 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:22.377632 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:22.377649 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:22.378463 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:22.378532 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:22.378561 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:22.378680 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.379190 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.379307 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:14:22.380463 22958 ts_tablet_manager.cc:1918] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:22.380801 22957 ts_tablet_manager.cc:1918] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:22.380885 22650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:22.381011 22957 log.cc:1196] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/c94dc08d8fc94d5893b6eb69cb2a60d2
I20260620 20:14:22.381230 22958 log.cc:1196] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/c94dc08d8fc94d5893b6eb69cb2a60d2
I20260620 20:14:22.381915 22958 ts_tablet_manager.cc:1939] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 07fc776b1ebf4d8b80487a723c651e55: Deleting consensus metadata
I20260620 20:14:22.382503 22956 ts_tablet_manager.cc:1905] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:22.383942 22456 catalog_manager.cc:4841] TS 07fc776b1ebf4d8b80487a723c651e55 (127.4.169.130:45263): tablet c94dc08d8fc94d5893b6eb69cb2a60d2 (table test-table [id=ffc33b70d122442f9dbbadc30b1296a7]) successfully deleted
I20260620 20:14:22.391319 22956 ts_tablet_manager.cc:1918] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:22.391546 22956 log.cc:1196] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/c94dc08d8fc94d5893b6eb69cb2a60d2
I20260620 20:14:22.392125 22956 ts_tablet_manager.cc:1939] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 2e2a693d42ff42d0bb725954ed347613: Deleting consensus metadata
I20260620 20:14:22.393986 22453 catalog_manager.cc:4841] TS 2e2a693d42ff42d0bb725954ed347613 (127.4.169.131:45163): tablet c94dc08d8fc94d5893b6eb69cb2a60d2 (table test-table [id=ffc33b70d122442f9dbbadc30b1296a7]) successfully deleted
I20260620 20:14:22.490238 22957 ts_tablet_manager.cc:1939] T c94dc08d8fc94d5893b6eb69cb2a60d2 P 76ea42dbdb2f4f379dba75bf8d0e2368: Deleting consensus metadata
I20260620 20:14:22.490430 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 22528
W20260620 20:14:22.508232 22454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.169.129:32957 (error 108)
I20260620 20:14:22.508994 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 22659
I20260620 20:14:22.510492 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.509491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.0.0.1:53214)
0620 20:14:22.510079 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.510236 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":753,"threads_started":1}
W20260620 20:14:22.510843 22454 catalog_manager.cc:4584] TS 76ea42dbdb2f4f379dba75bf8d0e2368 (127.4.169.129:32957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c94dc08d8fc94d5893b6eb69cb2a60d2: Network error: Client connection negotiation failed: client connection to 127.4.169.129:32957: connect: Connection refused (error 111)
I20260620 20:14:22.517359 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.510012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.4.169.130:45263)
0620 20:14:22.510193 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.510201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.510225 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:14:22.510421 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.510441 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.510562 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.510670 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:22.511211 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.512366 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.513782 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.514442 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.514570 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.515711 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:22.515722 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:22.515741 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:22.516534 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:22.516602 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:22.516640 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:22.516754 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.516893 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.517007 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:14:22.518045 22781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:22.564749 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.564194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.0.0.1:53216)
0620 20:14:22.564455 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.564567 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 20:14:22.565992 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.565551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.0.0.1:53230)
0620 20:14:22.565740 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.565828 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:14:22.608443 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 22659
I20260620 20:14:22.626370 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.625919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.0.0.1:53242)
0620 20:14:22.626127 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.626229 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 20:14:22.627552 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.627179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.0.0.1:53248)
0620 20:14:22.627329 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.627404 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 20:14:22.631898 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 22789
I20260620 20:14:22.639385 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.632666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.4.169.131:45163)
0620 20:14:22.632884 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.632889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.632967 (+ 78us) server_negotiation.cc:407] Connection header received
0620 20:14:22.633154 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.633158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.633241 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.633369 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:22.633836 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.634845 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.636169 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.636605 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.636751 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.637947 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:22.637956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:22.637970 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:22.638612 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:22.638661 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:22.638682 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:22.638798 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.638948 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.639047 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:14:22.640079 22911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:22.708279 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.707702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.0.0.1:53264)
0620 20:14:22.707973 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.708104 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 20:14:22.709393 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.708934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.0.0.1:53280)
0620 20:14:22.709141 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.709224 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 20:14:22.761361 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 22789
I20260620 20:14:22.780239 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 22438
I20260620 20:14:22.781350 22499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:22.890676 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.889197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:32957 (local address 127.0.0.1:53284)
0620 20:14:22.890137 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.890285 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.129:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":763}
I20260620 20:14:22.890865 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 22438
2026-06-20T20:14:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3014 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:22 dist-test-slave-4bwq krb5kdc[22968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:22 dist-test-slave-4bwq krb5kdc[22968](info): set up 2 sockets
Jun 20 20:14:22 dist-test-slave-4bwq krb5kdc[22968](info): commencing operation
krb5kdc: starting...
W20260620 20:14:23.986785 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:24 dist-test-slave-4bwq krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:14:24.229919 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:40753
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:24.558086 22981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:24.558367 22981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:24.558532 22981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:24.565686 22981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:24.565790 22981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:24.565837 22981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:24.565896 22981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:24.565943 22981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:24.575740 22981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:24.577008 22981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:24.578276 22981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:24.591396 22989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:24.591995 22986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:24.592172 22987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:24.592546 22981 server_base.cc:1034] running on GCE node
I20260620 20:14:24.593359 22981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:24.594945 22981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:24.596153 22981 hybrid_clock.cc:648] HybridClock initialized: now 1781986464596085 us; error 55 us; skew 500 ppm
I20260620 20:14:24.596513 22981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:24 dist-test-slave-4bwq krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:24.599778 22981 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:14:24.601300 22981 webserver.cc:460] Webserver started at http://127.4.169.190:43929/ using document root <none> and password file <none>
I20260620 20:14:24.602058 22981 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:24.602212 22981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:24.602589 22981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:24.605568 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "f8668e7dc35345ecb3992020831f1e33"
format_stamp: "Formatted at 2026-06-20 20:14:24 on dist-test-slave-4bwq"
server_key: "72beef0c388f85684b1f256a4ba78654"
server_key_iv: "5b22adcbda016264e1d97b25e659bd7b"
server_key_version: "encryptionkey@0"
I20260620 20:14:24.606317 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "f8668e7dc35345ecb3992020831f1e33"
format_stamp: "Formatted at 2026-06-20 20:14:24 on dist-test-slave-4bwq"
server_key: "72beef0c388f85684b1f256a4ba78654"
server_key_iv: "5b22adcbda016264e1d97b25e659bd7b"
server_key_version: "encryptionkey@0"
I20260620 20:14:24.611387 22981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 20:14:24.614863 22996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:24.616149 22981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:14:24.616356 22981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "f8668e7dc35345ecb3992020831f1e33"
format_stamp: "Formatted at 2026-06-20 20:14:24 on dist-test-slave-4bwq"
server_key: "72beef0c388f85684b1f256a4ba78654"
server_key_iv: "5b22adcbda016264e1d97b25e659bd7b"
server_key_version: "encryptionkey@0"
I20260620 20:14:24.616575 22981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:24.630209 22981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:24.634531 22981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:24.634806 22981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:24.653689 22981 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:40753
I20260620 20:14:24.653695 23047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:40753 every 8 connection(s)
I20260620 20:14:24.655167 22981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:24.659056 23048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:24.662971 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 22981
I20260620 20:14:24.663231 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:24.663806 4774 external_mini_cluster.cc:1428] Setting key 5894c52612a5af4261350f40618dac7e
I20260620 20:14:24.672369 23048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33: Bootstrap starting.
I20260620 20:14:24.676303 23048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:24.677675 23048 log.cc:824] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:14:24 dist-test-slave-4bwq krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:24.680814 23048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33: No bootstrap required, opened a new log
I20260620 20:14:24.687726 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:24.665748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.4.169.190:40753)
0620 20:14:24.666535 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:24.666547 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:24.666590 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:14:24.668217 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:24.668239 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:24.668656 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:24.668982 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:24.670875 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:24.672163 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:24.673987 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:24.674744 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:24.674880 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:24.680091 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:24.680101 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:24.680116 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:24.680152 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:24.683496 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:24.684201 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:24.684208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:24.684214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:24.684381 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:24.684781 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:24.684790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:24.684799 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:24.685336 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:24.685671 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:24.686071 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 20:14:24.686573 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":350,"threads_started":1}
I20260620 20:14:24.689071 23048 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8668e7dc35345ecb3992020831f1e33" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40753 } }
I20260620 20:14:24.689513 23048 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:24.689637 23048 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8668e7dc35345ecb3992020831f1e33, State: Initialized, Role: FOLLOWER
I20260620 20:14:24.690182 23048 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8668e7dc35345ecb3992020831f1e33" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40753 } }
I20260620 20:14:24.690464 23048 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:24.690635 23048 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:24.690834 23048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:24.692977 23048 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8668e7dc35345ecb3992020831f1e33" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40753 } }
I20260620 20:14:24.693446 23048 leader_election.cc:302] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8668e7dc35345ecb3992020831f1e33; no voters:
I20260620 20:14:24.694036 23048 leader_election.cc:288] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:24.694191 23053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:24.695652 23053 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [term 1 LEADER]: Becoming Leader. State: Replica: f8668e7dc35345ecb3992020831f1e33, State: Running, Role: LEADER
I20260620 20:14:24.696163 23048 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:24.696138 23053 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8668e7dc35345ecb3992020831f1e33" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40753 } }
I20260620 20:14:24.698985 23054 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8668e7dc35345ecb3992020831f1e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8668e7dc35345ecb3992020831f1e33" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40753 } } }
I20260620 20:14:24.699405 23054 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:24.699383 23055 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8668e7dc35345ecb3992020831f1e33. Latest consensus state: current_term: 1 leader_uuid: "f8668e7dc35345ecb3992020831f1e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8668e7dc35345ecb3992020831f1e33" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 40753 } } }
I20260620 20:14:24.699688 23055 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:24.702283 23061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:24.706527 23061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:24.715435 23061 catalog_manager.cc:1347] Generated new cluster ID: c61ae3f66c9549dab2ba287e13836216
I20260620 20:14:24.715560 23061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:24.729941 23061 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:24.731366 23061 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:24.739604 23061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33: Generated new TSK 0
I20260620 20:14:24.740490 23061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:14:24.806438 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40753
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:25.146229 23074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:25.146565 23074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:25.146733 23074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:25.154114 23074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:25.154254 23074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:25.154456 23074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:14:25.164202 23074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:25.165525 23074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:25.166898 23074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:25.179740 23080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:25.179840 23079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:25.180163 23082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:25.180820 23074 server_base.cc:1034] running on GCE node
I20260620 20:14:25.181453 23074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:25.182350 23074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:25.183624 23074 hybrid_clock.cc:648] HybridClock initialized: now 1781986465183574 us; error 41 us; skew 500 ppm
I20260620 20:14:25.183971 23074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:25 dist-test-slave-4bwq krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:25.187119 23074 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:14:25.188619 23074 webserver.cc:460] Webserver started at http://127.4.169.129:41333/ using document root <none> and password file <none>
I20260620 20:14:25.189263 23074 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:25.189359 23074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:25.189628 23074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:25.192458 23074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "86e89cadd0a641f8a45abed2a7d13ee4"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-4bwq"
server_key: "9811c05fe7a0a7186d66576c09b1ea73"
server_key_iv: "9e60f62c3e2a4fceb2ae4c1e416e0700"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.193231 23074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "86e89cadd0a641f8a45abed2a7d13ee4"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-4bwq"
server_key: "9811c05fe7a0a7186d66576c09b1ea73"
server_key_iv: "9e60f62c3e2a4fceb2ae4c1e416e0700"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.198941 23074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 20:14:25.202690 23089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.204231 23074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:25.204366 23074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "86e89cadd0a641f8a45abed2a7d13ee4"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-4bwq"
server_key: "9811c05fe7a0a7186d66576c09b1ea73"
server_key_iv: "9e60f62c3e2a4fceb2ae4c1e416e0700"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.204533 23074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:25.234964 23074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:25.239516 23074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:25.239842 23074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:25.240809 23074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:25.242143 23074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:25.242218 23074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.242301 23074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:25.242352 23074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.278632 23074 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:35099
I20260620 20:14:25.278679 23201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:35099 every 8 connection(s)
I20260620 20:14:25.279982 23074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:25.284639 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 23074
I20260620 20:14:25.284813 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:25.285280 4774 external_mini_cluster.cc:1428] Setting key b23bea75cd8a8d32474c7d46239bc059
Jun 20 20:14:25 dist-test-slave-4bwq krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:25.301066 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.282710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:49869 (local address 127.4.169.190:40753)
0620 20:14:25.283170 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.283174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.284165 (+ 991us) server_negotiation.cc:407] Connection header received
0620 20:14:25.286218 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.286223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.286316 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.286463 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:25.288070 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.288889 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.290420 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.290830 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.290975 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.294300 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:25.294305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:25.294308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:25.294337 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:25.297672 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.298373 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.298381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.298404 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.298568 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.298988 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.298995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.299001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.299380 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:25.299535 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.300241 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.300529 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260620 20:14:25.301723 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.283325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40753 (local address 127.4.169.129:49869)
0620 20:14:25.284015 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.284053 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:25.285923 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:25.286690 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:25.286698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:25.287360 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:25.287902 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.287918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.289048 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.289059 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.290256 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.290267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.291074 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.291084 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.291322 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.291997 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:25.292020 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:25.294089 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:25.297805 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:25.297811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:25.297823 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:25.298252 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:25.298687 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:25.298690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:25.298693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:25.298882 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:25.299508 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:25.299513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:25.299821 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 20:14:25.300075 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 20:14:25.300529 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":268,"threads_started":1}
I20260620 20:14:25.303301 23202 heartbeater.cc:347] Connected to a master server at 127.4.169.190:40753
I20260620 20:14:25.303620 23202 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:25.304322 23202 heartbeater.cc:511] Master 127.4.169.190:40753 requested a full tablet report, sending...
I20260620 20:14:25.306380 23013 ts_manager.cc:194] Registered new tserver with Master: 86e89cadd0a641f8a45abed2a7d13ee4 (127.4.169.129:35099)
I20260620 20:14:25.308053 23013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:49869
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:14:25.339677 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40753
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:25.667485 23208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:25.667743 23208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:25.667837 23208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:25.675068 23208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:25.675168 23208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:25.675280 23208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:14:25.684885 23208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:25.686034 23208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:25.687366 23208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:25.700840 23213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:25.700796 23214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:25.701062 23216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:25.701597 23208 server_base.cc:1034] running on GCE node
I20260620 20:14:25.702592 23208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:25.703593 23208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:25.704834 23208 hybrid_clock.cc:648] HybridClock initialized: now 1781986465704774 us; error 56 us; skew 500 ppm
I20260620 20:14:25.705168 23208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:25 dist-test-slave-4bwq krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:25.713013 23208 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:14:25.714818 23208 webserver.cc:460] Webserver started at http://127.4.169.130:43275/ using document root <none> and password file <none>
I20260620 20:14:25.715555 23208 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:25.715703 23208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:25.716033 23208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:25.719045 23208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "d089e6293fab4453987a107e8edf4b0f"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-4bwq"
server_key: "456e5bfcff0e70647ec72e0ab3f7ca4c"
server_key_iv: "ffff704cb5eebff1323a8bda3464d4e2"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.719843 23208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "d089e6293fab4453987a107e8edf4b0f"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-4bwq"
server_key: "456e5bfcff0e70647ec72e0ab3f7ca4c"
server_key_iv: "ffff704cb5eebff1323a8bda3464d4e2"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.724967 23208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:14:25.728441 23223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.729799 23208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:25.729992 23208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "d089e6293fab4453987a107e8edf4b0f"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-4bwq"
server_key: "456e5bfcff0e70647ec72e0ab3f7ca4c"
server_key_iv: "ffff704cb5eebff1323a8bda3464d4e2"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.730189 23208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:25.749140 23208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:25.753697 23208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:25.754014 23208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:25.755098 23208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:25.756559 23208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:25.756671 23208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.756829 23208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:25.756907 23208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.799088 23208 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:37019
I20260620 20:14:25.799192 23335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:37019 every 8 connection(s)
I20260620 20:14:25.800915 23208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:25.803870 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 23208
I20260620 20:14:25.804065 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:25.804709 4774 external_mini_cluster.cc:1428] Setting key 6f4471d6d5245a4e54ed042099dde066
Jun 20 20:14:25 dist-test-slave-4bwq krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:25.831096 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.804719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40753 (local address 127.4.169.130:50153)
0620 20:14:25.805551 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.805606 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:25.808205 (+ 2599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:25.809150 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:25.809159 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:25.809749 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:25.810325 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.810342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.811968 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.811976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.814042 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.814059 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.815524 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.815532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.815923 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.816828 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:25.816875 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:25.820510 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:25.825173 (+ 4663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:25.825184 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:25.825210 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:25.825945 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:25.826601 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:25.826607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:25.826614 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:25.826906 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:25.827787 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:25.827798 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:25.828359 (+ 561us) client_negotiation.cc:769] Sending connection context
0620 20:14:25.828712 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 20:14:25.829399 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":338,"threads_started":1}
I20260620 20:14:25.831357 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.804098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:50153 (local address 127.4.169.190:40753)
0620 20:14:25.804571 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.804578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.808551 (+ 3973us) server_negotiation.cc:407] Connection header received
0620 20:14:25.808619 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.808626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.808715 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.808819 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:25.810602 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.811773 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.814230 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.815012 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.815179 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.820876 (+ 5697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:25.820886 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:25.820892 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:25.820954 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:25.824921 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.826147 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.826155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.826159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.826345 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.827094 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.827102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.827106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.827530 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:25.827692 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.830659 (+ 2967us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.830934 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260620 20:14:25.832878 23336 heartbeater.cc:347] Connected to a master server at 127.4.169.190:40753
I20260620 20:14:25.833240 23336 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:14:25.834215 23336 heartbeater.cc:511] Master 127.4.169.190:40753 requested a full tablet report, sending...
I20260620 20:14:25.835950 23013 ts_manager.cc:194] Registered new tserver with Master: d089e6293fab4453987a107e8edf4b0f (127.4.169.130:37019)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
I20260620 20:14:25.836834 23013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:50153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:14:25.865875 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40753
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:26.189002 23342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:26.189276 23342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:26.189370 23342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:26.196703 23342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:26.196815 23342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:26.196981 23342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:14:26.206943 23342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:40753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:26.208324 23342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:26.209666 23342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:26.223266 23350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:26.223172 23347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:26.223227 23348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:26.224239 23342 server_base.cc:1034] running on GCE node
I20260620 20:14:26.224961 23342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:26.226033 23342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:26.227303 23342 hybrid_clock.cc:648] HybridClock initialized: now 1781986466227218 us; error 66 us; skew 500 ppm
I20260620 20:14:26.227643 23342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:26 dist-test-slave-4bwq krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:26.231153 23342 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:14:26.232659 23342 webserver.cc:460] Webserver started at http://127.4.169.131:35159/ using document root <none> and password file <none>
I20260620 20:14:26.233453 23342 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:26.233577 23342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:26.233873 23342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:26.236985 23342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "583ff246550e4ec1a831375adae46a7d"
format_stamp: "Formatted at 2026-06-20 20:14:26 on dist-test-slave-4bwq"
server_key: "edc37e3499dff8b7c060a41e7b0f00d8"
server_key_iv: "6e06a4ca8d642ba034e88cdf4a8d1a47"
server_key_version: "encryptionkey@0"
I20260620 20:14:26.237861 23342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "583ff246550e4ec1a831375adae46a7d"
format_stamp: "Formatted at 2026-06-20 20:14:26 on dist-test-slave-4bwq"
server_key: "edc37e3499dff8b7c060a41e7b0f00d8"
server_key_iv: "6e06a4ca8d642ba034e88cdf4a8d1a47"
server_key_version: "encryptionkey@0"
I20260620 20:14:26.243254 23342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:14:26.246868 23357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:26.248379 23342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:14:26.248522 23342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "583ff246550e4ec1a831375adae46a7d"
format_stamp: "Formatted at 2026-06-20 20:14:26 on dist-test-slave-4bwq"
server_key: "edc37e3499dff8b7c060a41e7b0f00d8"
server_key_iv: "6e06a4ca8d642ba034e88cdf4a8d1a47"
server_key_version: "encryptionkey@0"
I20260620 20:14:26.248699 23342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:26.261322 23342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:26.265401 23342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:26.265654 23342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:26.266611 23342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:26.267926 23342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:26.268011 23342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:26.268095 23342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:26.268139 23342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:26.301841 23342 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:37467
I20260620 20:14:26.301896 23469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:37467 every 8 connection(s)
I20260620 20:14:26.303298 23342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:26.309343 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 23342
I20260620 20:14:26.309545 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:26.310176 4774 external_mini_cluster.cc:1428] Setting key c7e9541eb3f5d29dea4a8e3451252af2
I20260620 20:14:26.311168 23202 heartbeater.cc:503] Master 127.4.169.190:40753 was elected leader, sending a full tablet report...
Jun 20 20:14:26 dist-test-slave-4bwq krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:26.325111 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.305983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:59593 (local address 127.4.169.190:40753)
0620 20:14:26.306176 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.306181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.307394 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 20:14:26.309536 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.309544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.309690 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.309797 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:26.311884 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.313031 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.314923 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.315335 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.315458 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.319054 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:26.319059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:26.319062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:26.319097 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:26.321953 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.322790 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.322794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.322796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.322932 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.323337 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.323340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.323342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.323642 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:26.323765 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.324523 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.324783 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:14:26.326301 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.306475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:40753 (local address 127.4.169.131:59593)
0620 20:14:26.307218 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:26.307267 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:26.309174 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:26.310077 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:26.310089 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:26.310816 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:26.311690 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.311717 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.313180 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.313186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.314793 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.314806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.315571 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.315577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.315770 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.316501 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:26.316525 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:26.318799 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:26.322094 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:26.322100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:26.322113 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:26.322647 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:26.323030 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:26.323033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:26.323035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:26.323227 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:26.323757 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:26.323762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:26.324122 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 20:14:26.324358 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 20:14:26.324881 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":290,"threads_started":1}
I20260620 20:14:26.327927 23470 heartbeater.cc:347] Connected to a master server at 127.4.169.190:40753
I20260620 20:14:26.328284 23470 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:26.329037 23470 heartbeater.cc:511] Master 127.4.169.190:40753 requested a full tablet report, sending...
I20260620 20:14:26.330605 23013 ts_manager.cc:194] Registered new tserver with Master: 583ff246550e4ec1a831375adae46a7d (127.4.169.131:37467)
I20260620 20:14:26.331429 23013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:59593
I20260620 20:14:26.339291 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:14:26.357647 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.343656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.4.169.190:40753)
0620 20:14:26.343905 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.343914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.344338 (+ 424us) server_negotiation.cc:407] Connection header received
0620 20:14:26.344436 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.344440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.344520 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.344601 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:26.346106 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.346949 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.348385 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.348811 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.348906 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.351756 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:26.351761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:26.351765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:26.351800 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:26.354700 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.355509 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.355514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.355517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.355653 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.356123 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.356132 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.356134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.356448 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:26.356742 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.357126 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.357321 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:14:26.361994 23013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:26.368919 23013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:26.392316 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.382182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.4.169.130:37019)
0620 20:14:26.383766 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.383776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.383810 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:14:26.383925 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.383933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.384209 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.385146 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:26.385313 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.386299 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.389276 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.390577 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.390725 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.390951 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.391730 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.391901 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1398,"thread_start_us":1263,"threads_started":1}
I20260620 20:14:26.392431 23482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.381718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:37019 (local address 127.0.0.1:40298)
0620 20:14:26.382883 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:26.382928 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:26.383095 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:26.384488 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:26.384495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:26.384530 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:26.384980 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.384990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.386493 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.386502 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.389096 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.389107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.391250 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.391259 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.391489 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.391531 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 20:14:26.391680 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:14:26.391899 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":1,"spinlock_wait_cycles":7168,"thread_start_us":838,"threads_started":1}
I20260620 20:14:26.395591 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.382668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37467 (local address 127.0.0.1:40270)
0620 20:14:26.386993 (+ 4325us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:26.387022 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:26.387230 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:26.388204 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:26.388209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:26.388234 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:26.388693 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.388705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.390841 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.390851 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.393239 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.393257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.394719 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.394726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.394896 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.394950 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 20:14:26.395063 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:14:26.395193 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4175,"thread_start_us":1610,"threads_started":1}
I20260620 20:14:26.396199 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.383352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.4.169.131:37467)
0620 20:14:26.384127 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.384135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.387123 (+ 2988us) server_negotiation.cc:407] Connection header received
0620 20:14:26.387408 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.387418 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.387672 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.387831 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:26.388899 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.390537 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.393421 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.394517 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.394660 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.394905 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.395108 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.395268 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":455,"threads_started":1}
I20260620 20:14:26.396872 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.382032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:35099 (local address 127.0.0.1:36290)
0620 20:14:26.384534 (+ 2502us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:26.384556 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:26.384683 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:26.387666 (+ 2983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:26.387670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:26.387697 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:26.388249 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.388264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.390604 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.390609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.393670 (+ 3061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.393681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.395606 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.395613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.395775 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.395809 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:14:26.395877 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:14:26.396016 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2389,"mutex_wait_us":423,"spinlock_wait_cycles":1280,"thread_start_us":4832,"threads_started":1}
I20260620 20:14:26.396979 23271 tablet_service.cc:1467] Processing CreateTablet for tablet 69f4553904634dfe8e9e3300d54343ce (DEFAULT_TABLE table=test-table [id=a2d69a3d8073437d8857ee8aa4c6dcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:26.397928 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.382622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.4.169.129:35099)
0620 20:14:26.387016 (+ 4394us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.387026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.387058 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:14:26.387162 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.387171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.387442 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.388481 (+ 1039us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:26.388728 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.390430 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.393891 (+ 3461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.395358 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.397128 (+ 1770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.397370 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.397437 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.397605 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4187,"thread_start_us":4026,"threads_started":1}
I20260620 20:14:26.398634 23271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f4553904634dfe8e9e3300d54343ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:26.400125 23405 tablet_service.cc:1467] Processing CreateTablet for tablet 69f4553904634dfe8e9e3300d54343ce (DEFAULT_TABLE table=test-table [id=a2d69a3d8073437d8857ee8aa4c6dcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:26.400957 23137 tablet_service.cc:1467] Processing CreateTablet for tablet 69f4553904634dfe8e9e3300d54343ce (DEFAULT_TABLE table=test-table [id=a2d69a3d8073437d8857ee8aa4c6dcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:26.402139 23405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f4553904634dfe8e9e3300d54343ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:26.402242 23137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f4553904634dfe8e9e3300d54343ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:26.413843 23488 tablet_bootstrap.cc:492] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: Bootstrap starting.
I20260620 20:14:26.417572 23488 tablet_bootstrap.cc:654] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:26.418087 23489 tablet_bootstrap.cc:492] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: Bootstrap starting.
I20260620 20:14:26.419526 23490 tablet_bootstrap.cc:492] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: Bootstrap starting.
I20260620 20:14:26.419557 23488 log.cc:824] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:26.422921 23490 tablet_bootstrap.cc:654] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:26.423100 23489 tablet_bootstrap.cc:654] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:26.423311 23488 tablet_bootstrap.cc:492] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: No bootstrap required, opened a new log
I20260620 20:14:26.423633 23488 ts_tablet_manager.cc:1397] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 20:14:26.424147 23490 log.cc:824] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:26.424361 23489 log.cc:824] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:26.427168 23489 tablet_bootstrap.cc:492] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: No bootstrap required, opened a new log
I20260620 20:14:26.427436 23490 tablet_bootstrap.cc:492] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: No bootstrap required, opened a new log
I20260620 20:14:26.427493 23489 ts_tablet_manager.cc:1397] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 20:14:26.427706 23490 ts_tablet_manager.cc:1397] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260620 20:14:26.434551 23488 raft_consensus.cc:348] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.435043 23488 raft_consensus.cc:374] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:26.435201 23488 raft_consensus.cc:729] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e89cadd0a641f8a45abed2a7d13ee4, State: Initialized, Role: FOLLOWER
I20260620 20:14:26.435387 23489 raft_consensus.cc:348] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.435799 23489 raft_consensus.cc:374] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:26.435904 23489 raft_consensus.cc:729] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d089e6293fab4453987a107e8edf4b0f, State: Initialized, Role: FOLLOWER
I20260620 20:14:26.435863 23488 consensus_queue.cc:260] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.436508 23489 consensus_queue.cc:260] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.437711 23488 ts_tablet_manager.cc:1428] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 20:14:26.438228 23490 raft_consensus.cc:348] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.438750 23490 raft_consensus.cc:374] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:26.438745 23336 heartbeater.cc:503] Master 127.4.169.190:40753 was elected leader, sending a full tablet report...
I20260620 20:14:26.438925 23490 raft_consensus.cc:729] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583ff246550e4ec1a831375adae46a7d, State: Initialized, Role: FOLLOWER
I20260620 20:14:26.439374 23489 ts_tablet_manager.cc:1428] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260620 20:14:26.439657 23490 consensus_queue.cc:260] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.441438 23470 heartbeater.cc:503] Master 127.4.169.190:40753 was elected leader, sending a full tablet report...
I20260620 20:14:26.442073 23490 ts_tablet_manager.cc:1428] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260620 20:14:26.503304 23495 raft_consensus.cc:482] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:26.503654 23495 raft_consensus.cc:504] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.505511 23495 leader_election.cc:288] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86e89cadd0a641f8a45abed2a7d13ee4 (127.4.169.129:35099), 583ff246550e4ec1a831375adae46a7d (127.4.169.131:37467)
I20260620 20:14:26.507985 23496 raft_consensus.cc:482] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:26.508289 23496 raft_consensus.cc:504] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.510522 23496 leader_election.cc:288] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d089e6293fab4453987a107e8edf4b0f (127.4.169.130:37019), 86e89cadd0a641f8a45abed2a7d13ee4 (127.4.169.129:35099)
I20260620 20:14:26.514678 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.506520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:47613 (local address 127.4.169.131:37467)
0620 20:14:26.506738 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.506744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.507028 (+ 284us) server_negotiation.cc:407] Connection header received
0620 20:14:26.507240 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.507245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.507328 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.507427 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:26.508435 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.509463 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.512173 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.513466 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.513582 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.513659 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.514242 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.514413 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:14:26.515621 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.505791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37467 (local address 127.4.169.130:47613)
0620 20:14:26.506849 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:26.506876 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:26.507098 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:26.507493 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:26.507499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:26.507538 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:26.508288 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.508299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.509646 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.509651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.511988 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.512004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.513859 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.513865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.514031 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.514068 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:14:26.515078 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0620 20:14:26.515231 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":48,"spinlock_wait_cycles":37120,"thread_start_us":400,"threads_started":1}
I20260620 20:14:26.515657 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.506008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:36391 (local address 127.4.169.129:35099)
0620 20:14:26.506282 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.506292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.507021 (+ 729us) server_negotiation.cc:407] Connection header received
0620 20:14:26.507260 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.507265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.507334 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.507427 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:26.508326 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.509444 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.512263 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.514034 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.514219 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.514328 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.515111 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.515302 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:14:26.516713 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.505922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:35099 (local address 127.4.169.130:36391)
0620 20:14:26.506866 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:26.506880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:26.507059 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:26.507493 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:26.507499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:26.507546 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:26.508168 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.508185 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.509590 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.509596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.512032 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.512050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.514603 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.514611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.514860 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.514911 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 20:14:26.516055 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0620 20:14:26.516200 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":155,"spinlock_wait_cycles":6528,"thread_start_us":643,"threads_started":1}
I20260620 20:14:26.516846 23425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f4553904634dfe8e9e3300d54343ce" candidate_uuid: "d089e6293fab4453987a107e8edf4b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583ff246550e4ec1a831375adae46a7d" is_pre_election: true
I20260620 20:14:26.517310 23425 raft_consensus.cc:2454] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d089e6293fab4453987a107e8edf4b0f in term 0.
I20260620 20:14:26.518280 23227 leader_election.cc:302] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583ff246550e4ec1a831375adae46a7d, d089e6293fab4453987a107e8edf4b0f; no voters:
I20260620 20:14:26.518862 23495 raft_consensus.cc:2781] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:26.519043 23495 raft_consensus.cc:482] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:26.519163 23495 raft_consensus.cc:3037] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:26.519184 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.510854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:37019 (local address 127.4.169.131:52611)
0620 20:14:26.511145 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:26.511166 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:26.511265 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:26.512193 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:26.512197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:26.512233 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:26.512461 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.512472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.513808 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.513817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.515786 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.515819 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.517591 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.517600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.517912 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.517968 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 20:14:26.518092 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 20:14:26.518282 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 20:14:26.523674 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f4553904634dfe8e9e3300d54343ce" candidate_uuid: "d089e6293fab4453987a107e8edf4b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" is_pre_election: true
I20260620 20:14:26.524236 23157 raft_consensus.cc:2454] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d089e6293fab4453987a107e8edf4b0f in term 0.
I20260620 20:14:26.526458 23495 raft_consensus.cc:504] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.528362 23157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f4553904634dfe8e9e3300d54343ce" candidate_uuid: "d089e6293fab4453987a107e8edf4b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86e89cadd0a641f8a45abed2a7d13ee4"
I20260620 20:14:26.528695 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.511291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:35099 (local address 127.4.169.131:48125)
0620 20:14:26.519606 (+ 8315us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:26.519630 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:26.519754 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:26.520432 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:26.520440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:26.520474 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:26.520714 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.520728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.522751 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.522759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.524776 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:26.524796 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.527583 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.527594 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:26.527898 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.527954 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 20:14:26.528078 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 20:14:26.528280 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8184,"threads_started":1}
I20260620 20:14:26.528643 23157 raft_consensus.cc:3037] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:26.527064 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.511604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:52611 (local address 127.4.169.130:37019)
0620 20:14:26.511763 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.511770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.511787 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:14:26.511844 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.511850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.511935 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.512029 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:26.512635 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.513647 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.516134 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.517401 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.517554 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.517628 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.518484 (+ 856us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.518668 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:14:26.529520 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.511843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:48125 (local address 127.4.169.129:35099)
0620 20:14:26.515937 (+ 4094us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.515946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.519883 (+ 3937us) server_negotiation.cc:407] Connection header received
0620 20:14:26.519965 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.519973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.520087 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.520183 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:26.520903 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.522472 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.525023 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.526948 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.527163 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.527275 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.528908 (+ 1633us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.529142 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3966,"thread_start_us":17335,"threads_started":1}
I20260620 20:14:26.529911 23495 leader_election.cc:288] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [CANDIDATE]: Term 1 election: Requested vote from peers 86e89cadd0a641f8a45abed2a7d13ee4 (127.4.169.129:35099), 583ff246550e4ec1a831375adae46a7d (127.4.169.131:37467)
I20260620 20:14:26.530309 23156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f4553904634dfe8e9e3300d54343ce" candidate_uuid: "583ff246550e4ec1a831375adae46a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" is_pre_election: true
I20260620 20:14:26.530534 23425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f4553904634dfe8e9e3300d54343ce" candidate_uuid: "d089e6293fab4453987a107e8edf4b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "583ff246550e4ec1a831375adae46a7d"
I20260620 20:14:26.530519 23291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f4553904634dfe8e9e3300d54343ce" candidate_uuid: "583ff246550e4ec1a831375adae46a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d089e6293fab4453987a107e8edf4b0f" is_pre_election: true
I20260620 20:14:26.530746 23425 raft_consensus.cc:3037] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:26.531052 23291 raft_consensus.cc:2379] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 583ff246550e4ec1a831375adae46a7d in current term 1: Already voted for candidate d089e6293fab4453987a107e8edf4b0f in this term.
I20260620 20:14:26.531945 23360 leader_election.cc:302] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 583ff246550e4ec1a831375adae46a7d; no voters: 86e89cadd0a641f8a45abed2a7d13ee4, d089e6293fab4453987a107e8edf4b0f
W20260620 20:14:26.532428 23203 tablet.cc:2369] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:26.532459 23157 raft_consensus.cc:2454] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d089e6293fab4453987a107e8edf4b0f in term 1.
I20260620 20:14:26.533025 23225 leader_election.cc:302] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86e89cadd0a641f8a45abed2a7d13ee4, d089e6293fab4453987a107e8edf4b0f; no voters:
I20260620 20:14:26.533322 23495 raft_consensus.cc:2781] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:26.533797 23495 raft_consensus.cc:686] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 1 LEADER]: Becoming Leader. State: Replica: d089e6293fab4453987a107e8edf4b0f, State: Running, Role: LEADER
I20260620 20:14:26.534343 23495 consensus_queue.cc:237] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } }
I20260620 20:14:26.534744 23425 raft_consensus.cc:2454] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d089e6293fab4453987a107e8edf4b0f in term 1.
I20260620 20:14:26.534931 23496 raft_consensus.cc:2726] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 20:14:26.539229 23012 catalog_manager.cc:5495] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to d089e6293fab4453987a107e8edf4b0f (127.4.169.130). New cstate: current_term: 1 leader_uuid: "d089e6293fab4453987a107e8edf4b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d089e6293fab4453987a107e8edf4b0f" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 37019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 } health_report { overall_health: UNKNOWN } } }
W20260620 20:14:26.553303 23337 tablet.cc:2369] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:26.555050 23471 tablet.cc:2369] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:26.566105 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.560835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.4.169.130:37019)
0620 20:14:26.561147 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.561152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.561176 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:14:26.561303 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.561307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.561380 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.561498 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:26.561840 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.562674 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.563966 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.564367 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.564465 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.564959 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:26.565447 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:26.565575 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.565715 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.565784 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260620 20:14:26.573974 23157 raft_consensus.cc:1261] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [term 1 FOLLOWER]: Refusing update from remote peer d089e6293fab4453987a107e8edf4b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:26.574023 23425 raft_consensus.cc:1261] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 1 FOLLOWER]: Refusing update from remote peer d089e6293fab4453987a107e8edf4b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:26.574919 23501 consensus_queue.cc:1035] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "86e89cadd0a641f8a45abed2a7d13ee4" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:26.575286 23495 consensus_queue.cc:1035] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "583ff246550e4ec1a831375adae46a7d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:26.586236 23504 mvcc.cc:205] Tried to move back new op lower bound from 7299016567077535744 to 7299016566929473536. Current Snapshot: MvccSnapshot[applied={T|T < 7299016567077535744}]
I20260620 20:14:26.590988 23505 mvcc.cc:205] Tried to move back new op lower bound from 7299016567077535744 to 7299016566929473536. Current Snapshot: MvccSnapshot[applied={T|T < 7299016567077535744}]
I20260620 20:14:26.604907 23013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:26.605232 23013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:26.609208 23013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8668e7dc35345ecb3992020831f1e33: Sending DeleteTablet for 3 replicas of tablet 69f4553904634dfe8e9e3300d54343ce
I20260620 20:14:26.610334 23137 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f4553904634dfe8e9e3300d54343ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:26 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:36290
I20260620 20:14:26.610455 23271 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f4553904634dfe8e9e3300d54343ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:26 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:40298
I20260620 20:14:26.610455 23405 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f4553904634dfe8e9e3300d54343ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:26 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:40270
I20260620 20:14:26.611310 23514 tablet_replica.cc:330] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: stopping tablet replica
I20260620 20:14:26.611775 23514 raft_consensus.cc:2229] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:26.612370 23514 raft_consensus.cc:2258] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:26.614766 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 23074
I20260620 20:14:26.622690 23514 ts_tablet_manager.cc:1905] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:26.629612 23515 tablet_replica.cc:330] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: stopping tablet replica
I20260620 20:14:26.633792 23515 raft_consensus.cc:2229] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:26.636116 23515 raft_consensus.cc:2258] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:26.640312 23515 ts_tablet_manager.cc:1905] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:26.641021 23516 tablet_replica.cc:330] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: stopping tablet replica
I20260620 20:14:26.641418 23516 raft_consensus.cc:2229] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:26.641875 23516 raft_consensus.cc:2258] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:26.644029 23514 ts_tablet_manager.cc:1918] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:26.644222 23514 log.cc:1196] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/69f4553904634dfe8e9e3300d54343ce
I20260620 20:14:26.644696 23514 ts_tablet_manager.cc:1939] T 69f4553904634dfe8e9e3300d54343ce P 86e89cadd0a641f8a45abed2a7d13ee4: Deleting consensus metadata
Jun 20 20:14:26 dist-test-slave-4bwq krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:14:26.646443 22998 catalog_manager.cc:4841] TS 86e89cadd0a641f8a45abed2a7d13ee4 (127.4.169.129:35099): tablet 69f4553904634dfe8e9e3300d54343ce (table test-table [id=a2d69a3d8073437d8857ee8aa4c6dcc2]) successfully deleted
I20260620 20:14:26.651257 23516 ts_tablet_manager.cc:1905] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:26.654526 23515 ts_tablet_manager.cc:1918] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:26.654704 23515 log.cc:1196] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/69f4553904634dfe8e9e3300d54343ce
I20260620 20:14:26.654997 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.616473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.4.169.129:35099)
0620 20:14:26.616709 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.616719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.635081 (+ 18362us) server_negotiation.cc:407] Connection header received
0620 20:14:26.635311 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.635317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.635430 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.635554 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:26.637352 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.638540 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.640528 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.641136 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.641235 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.647020 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:26.647035 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:26.647054 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:26.647096 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:26.651070 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.652047 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.652058 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.652062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.652216 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.652802 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.652810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.652815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.653355 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:26.654030 (+ 675us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.654196 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.654609 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 20:14:26.655668 23515 ts_tablet_manager.cc:1939] T 69f4553904634dfe8e9e3300d54343ce P d089e6293fab4453987a107e8edf4b0f: Deleting consensus metadata
I20260620 20:14:26.656895 23197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:26.657438 22999 catalog_manager.cc:4841] TS d089e6293fab4453987a107e8edf4b0f (127.4.169.130:37019): tablet 69f4553904634dfe8e9e3300d54343ce (table test-table [id=a2d69a3d8073437d8857ee8aa4c6dcc2]) successfully deleted
I20260620 20:14:26.658145 23516 ts_tablet_manager.cc:1918] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:26.658313 23516 log.cc:1196] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/69f4553904634dfe8e9e3300d54343ce
I20260620 20:14:26.658835 23516 ts_tablet_manager.cc:1939] T 69f4553904634dfe8e9e3300d54343ce P 583ff246550e4ec1a831375adae46a7d: Deleting consensus metadata
I20260620 20:14:26.660637 23000 catalog_manager.cc:4841] TS 583ff246550e4ec1a831375adae46a7d (127.4.169.131:37467): tablet 69f4553904634dfe8e9e3300d54343ce (table test-table [id=a2d69a3d8073437d8857ee8aa4c6dcc2]) successfully deleted
I20260620 20:14:26.752611 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 23074
I20260620 20:14:26.771027 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 23208
Jun 20 20:14:26 dist-test-slave-4bwq krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:14:26.787901 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.771880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.4.169.130:37019)
0620 20:14:26.772108 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.772114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.772132 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:14:26.772310 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.772316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.772389 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.772509 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:26.773827 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.774691 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.776053 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.776498 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.776644 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.781073 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:26.781092 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:26.781107 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:26.781156 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:26.784592 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.785483 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.785490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.785492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.785644 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.786050 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.786056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.786061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.786638 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:26.786966 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.787299 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.787550 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:14:26.788672 23331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:26.902045 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 23208
I20260620 20:14:26.920869 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 23342
Jun 20 20:14:26 dist-test-slave-4bwq krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:14:26.936825 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:26.921627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.4.169.131:37467)
0620 20:14:26.921811 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:26.921816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:26.921881 (+ 65us) server_negotiation.cc:407] Connection header received
0620 20:14:26.922082 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:26.922088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:26.922170 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:26.922266 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:26.923745 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.924483 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.925806 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:26.926215 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:26.926312 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:26.930951 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:26.930972 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:26.930993 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:26.931034 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:26.933968 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.934796 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.934804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.934806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.934934 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:26.935347 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:26.935351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:26.935353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:26.935747 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:26.935999 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:26.936347 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 20:14:26.936553 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:14:26.937542 23465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:27.029389 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 23342
I20260620 20:14:27.050956 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 22981
I20260620 20:14:27.052116 23043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:27.163123 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 22981
2026-06-20T20:14:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4277 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T20:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:27Z Disabled control of system clock
I20260620 20:14:27.237849 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34877
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:14:27.570608 23525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:27.570891 23525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:27.570981 23525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:27.578153 23525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:27.578264 23525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:27.578333 23525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:27.578451 23525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:27.578522 23525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:27.587949 23525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:27.589284 23525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:27.591095 23525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:27.604238 23533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:27.604234 23531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:27.604759 23530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:27.604946 23525 server_base.cc:1034] running on GCE node
I20260620 20:14:27.605832 23525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:27.607306 23525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:27.608510 23525 hybrid_clock.cc:648] HybridClock initialized: now 1781986467608457 us; error 35 us; skew 500 ppm
I20260620 20:14:27.608867 23525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:27.611145 23525 webserver.cc:460] Webserver started at http://127.4.169.190:46345/ using document root <none> and password file <none>
I20260620 20:14:27.611815 23525 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:27.611903 23525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:27.612222 23525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:27.615216 23525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "4cf46d6a87e24aaeaf809d69a0a780ca"
format_stamp: "Formatted at 2026-06-20 20:14:27 on dist-test-slave-4bwq"
server_key: "4a26df2a3e3b5fa7003f914915a0c62c"
server_key_iv: "e864f6c0e7c3fc7cdafd9a87a488a14d"
server_key_version: "encryptionkey@0"
I20260620 20:14:27.616017 23525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "4cf46d6a87e24aaeaf809d69a0a780ca"
format_stamp: "Formatted at 2026-06-20 20:14:27 on dist-test-slave-4bwq"
server_key: "4a26df2a3e3b5fa7003f914915a0c62c"
server_key_iv: "e864f6c0e7c3fc7cdafd9a87a488a14d"
server_key_version: "encryptionkey@0"
I20260620 20:14:27.621057 23525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:14:27.624755 23539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:27.626173 23525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:27.626370 23525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "4cf46d6a87e24aaeaf809d69a0a780ca"
format_stamp: "Formatted at 2026-06-20 20:14:27 on dist-test-slave-4bwq"
server_key: "4a26df2a3e3b5fa7003f914915a0c62c"
server_key_iv: "e864f6c0e7c3fc7cdafd9a87a488a14d"
server_key_version: "encryptionkey@0"
I20260620 20:14:27.626663 23525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:27.671981 23525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:27.672883 23525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:27.673198 23525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:27.692633 23525 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:34877
I20260620 20:14:27.692670 23590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:34877 every 8 connection(s)
I20260620 20:14:27.694232 23525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:27.697994 23591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:27.700839 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 23525
I20260620 20:14:27.701035 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:27.701576 4774 external_mini_cluster.cc:1428] Setting key 600cf5001411758d2a15bb633f8aec06
I20260620 20:14:27.710562 23591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca: Bootstrap starting.
I20260620 20:14:27.714289 23591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:27.715376 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:27.703558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.4.169.190:34877)
0620 20:14:27.704695 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:27.704705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:27.704745 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:14:27.706484 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:27.706508 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:27.706989 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:27.707327 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:27.707924 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:27.709207 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:27.711003 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:27.711674 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:27.711814 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:27.713066 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:27.713075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:27.713088 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:27.713613 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:27.713687 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:27.713708 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:27.713799 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:27.714018 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:14:27.714339 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":487,"threads_started":1}
I20260620 20:14:27.715374 23591 log.cc:824] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:27.718531 23591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca: No bootstrap required, opened a new log
I20260620 20:14:27.725857 23591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf46d6a87e24aaeaf809d69a0a780ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34877 } }
I20260620 20:14:27.726253 23591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:27.726364 23591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf46d6a87e24aaeaf809d69a0a780ca, State: Initialized, Role: FOLLOWER
I20260620 20:14:27.727037 23591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf46d6a87e24aaeaf809d69a0a780ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34877 } }
I20260620 20:14:27.727283 23591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:27.727427 23591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:27.727609 23591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:27.730077 23591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf46d6a87e24aaeaf809d69a0a780ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34877 } }
I20260620 20:14:27.730708 23591 leader_election.cc:302] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cf46d6a87e24aaeaf809d69a0a780ca; no voters:
I20260620 20:14:27.731338 23591 leader_election.cc:288] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:27.731474 23596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:27.732193 23596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [term 1 LEADER]: Becoming Leader. State: Replica: 4cf46d6a87e24aaeaf809d69a0a780ca, State: Running, Role: LEADER
I20260620 20:14:27.732748 23596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf46d6a87e24aaeaf809d69a0a780ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34877 } }
I20260620 20:14:27.733347 23591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:27.735872 23597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cf46d6a87e24aaeaf809d69a0a780ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf46d6a87e24aaeaf809d69a0a780ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34877 } } }
I20260620 20:14:27.736325 23597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:27.736946 23598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cf46d6a87e24aaeaf809d69a0a780ca. Latest consensus state: current_term: 1 leader_uuid: "4cf46d6a87e24aaeaf809d69a0a780ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf46d6a87e24aaeaf809d69a0a780ca" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34877 } } }
I20260620 20:14:27.737311 23598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:27.739347 23604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:27.745448 23604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:27.754050 23604 catalog_manager.cc:1347] Generated new cluster ID: 55279f76a36949c1a8cf66ad249b111f
I20260620 20:14:27.754173 23604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:27.768282 23604 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:27.770149 23604 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:27.781409 23604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca: Generated new TSK 0
I20260620 20:14:27.782423 23604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:27.788956 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34877
--builtin_ntp_servers=127.4.169.148:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:14:28.121975 23615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:28.122262 23615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:28.122357 23615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:28.129803 23615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:28.129941 23615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:28.130138 23615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:14:28.139847 23615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:28.141260 23615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:28.143185 23615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:28.156112 23623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:28.156270 23621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:28.156513 23620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:28.157363 23615 server_base.cc:1034] running on GCE node
I20260620 20:14:28.158000 23615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:28.159014 23615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:28.160251 23615 hybrid_clock.cc:648] HybridClock initialized: now 1781986468160199 us; error 49 us; skew 500 ppm
I20260620 20:14:28.160567 23615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:28.162833 23615 webserver.cc:460] Webserver started at http://127.4.169.129:43603/ using document root <none> and password file <none>
I20260620 20:14:28.163534 23615 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:28.163664 23615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:28.163981 23615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:28.166867 23615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "1c8aef393522453698157a20594a9641"
format_stamp: "Formatted at 2026-06-20 20:14:28 on dist-test-slave-4bwq"
server_key: "95dddade3b4f37b20246b15647b39f6d"
server_key_iv: "c20590ec581867be18563047e4f8df19"
server_key_version: "encryptionkey@0"
I20260620 20:14:28.167685 23615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "1c8aef393522453698157a20594a9641"
format_stamp: "Formatted at 2026-06-20 20:14:28 on dist-test-slave-4bwq"
server_key: "95dddade3b4f37b20246b15647b39f6d"
server_key_iv: "c20590ec581867be18563047e4f8df19"
server_key_version: "encryptionkey@0"
I20260620 20:14:28.173180 23615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 20:14:28.176657 23629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:28.177968 23615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:28.178138 23615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "1c8aef393522453698157a20594a9641"
format_stamp: "Formatted at 2026-06-20 20:14:28 on dist-test-slave-4bwq"
server_key: "95dddade3b4f37b20246b15647b39f6d"
server_key_iv: "c20590ec581867be18563047e4f8df19"
server_key_version: "encryptionkey@0"
I20260620 20:14:28.178357 23615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:28.197897 23615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:28.198776 23615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:28.199090 23615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:28.200074 23615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:28.201543 23615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:28.201674 23615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:28.201804 23615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:28.201893 23615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:28.235502 23615 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:40071
I20260620 20:14:28.235594 23741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:40071 every 8 connection(s)
I20260620 20:14:28.236819 23615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:28.243100 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 23615
I20260620 20:14:28.243306 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:28.243858 4774 external_mini_cluster.cc:1428] Setting key bff7f0f411651d98286c9b7c6d99b547
I20260620 20:14:28.246901 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34877
--builtin_ntp_servers=127.4.169.148:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:14:28.250857 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:28.239502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:50727 (local address 127.4.169.190:34877)
0620 20:14:28.239968 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:28.239973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:28.241096 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 20:14:28.243104 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:28.243112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:28.243201 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:28.243320 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:28.244739 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:28.245525 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:28.247946 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:28.248360 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:28.248459 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:28.250105 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:28.250111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:28.250114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:28.250130 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:28.250171 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:28.250183 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:28.250253 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:28.250536 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 20:14:28.250616 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":253,"threads_started":1}
I20260620 20:14:28.252344 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:28.239917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34877 (local address 127.4.169.129:50727)
0620 20:14:28.240941 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:28.240983 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:28.242832 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:28.243604 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:28.243616 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:28.243733 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:28.244535 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:28.244556 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:28.245686 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:28.245690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:28.247796 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:28.247815 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:28.248594 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:28.248603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:28.248932 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:28.249746 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:28.249752 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:28.249806 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:28.249914 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:28.250281 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:28.250293 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:28.250304 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:14:28.250513 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 20:14:28.250938 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":290,"threads_started":1}
I20260620 20:14:28.253891 23742 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34877
I20260620 20:14:28.254212 23742 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:28.255024 23742 heartbeater.cc:511] Master 127.4.169.190:34877 requested a full tablet report, sending...
I20260620 20:14:28.257155 23556 ts_manager.cc:194] Registered new tserver with Master: 1c8aef393522453698157a20594a9641 (127.4.169.129:40071)
I20260620 20:14:28.259354 23556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.129:50727
W20260620 20:14:28.578298 23746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:28.578600 23746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:28.578687 23746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:28.586115 23746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:28.586221 23746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:28.586329 23746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:14:28.595675 23746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:28.596942 23746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:28.598780 23746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:28.613122 23752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:28.613153 23754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:28.613299 23751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:28.613991 23746 server_base.cc:1034] running on GCE node
I20260620 20:14:28.614656 23746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:28.615610 23746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:28.616838 23746 hybrid_clock.cc:648] HybridClock initialized: now 1781986468616804 us; error 63 us; skew 500 ppm
I20260620 20:14:28.617159 23746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:28.619467 23746 webserver.cc:460] Webserver started at http://127.4.169.130:33365/ using document root <none> and password file <none>
I20260620 20:14:28.620174 23746 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:28.620304 23746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:28.620640 23746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:28.623612 23746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "fe25cd52b2e94d2fa8f4c3356586ed70"
format_stamp: "Formatted at 2026-06-20 20:14:28 on dist-test-slave-4bwq"
server_key: "e327c1325a03406ad6621816d30db462"
server_key_iv: "65c196f8f8af2b2a7539b9580ca3d5c1"
server_key_version: "encryptionkey@0"
I20260620 20:14:28.624382 23746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "fe25cd52b2e94d2fa8f4c3356586ed70"
format_stamp: "Formatted at 2026-06-20 20:14:28 on dist-test-slave-4bwq"
server_key: "e327c1325a03406ad6621816d30db462"
server_key_iv: "65c196f8f8af2b2a7539b9580ca3d5c1"
server_key_version: "encryptionkey@0"
I20260620 20:14:28.629921 23746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 20:14:28.633558 23760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:28.635041 23746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:14:28.635210 23746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "fe25cd52b2e94d2fa8f4c3356586ed70"
format_stamp: "Formatted at 2026-06-20 20:14:28 on dist-test-slave-4bwq"
server_key: "e327c1325a03406ad6621816d30db462"
server_key_iv: "65c196f8f8af2b2a7539b9580ca3d5c1"
server_key_version: "encryptionkey@0"
I20260620 20:14:28.635424 23746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:28.656605 23746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:28.657469 23746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:28.657792 23746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:28.658902 23746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:28.660409 23746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:28.660547 23746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:28.660671 23746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:28.660758 23746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:28.695111 23746 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:43577
I20260620 20:14:28.695168 23872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:43577 every 8 connection(s)
I20260620 20:14:28.696494 23746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:28.704980 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 23746
I20260620 20:14:28.705163 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:28.705719 4774 external_mini_cluster.cc:1428] Setting key c90deb1870296a40fc48323cf9279e48
I20260620 20:14:28.708621 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34877
--builtin_ntp_servers=127.4.169.148:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:14:28.710842 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:28.699263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:43609 (local address 127.4.169.190:34877)
0620 20:14:28.699440 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:28.699446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:28.701031 (+ 1585us) server_negotiation.cc:407] Connection header received
0620 20:14:28.702938 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:28.702950 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:28.703041 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:28.703154 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:28.704270 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:28.705015 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:28.707041 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:28.707728 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:28.707873 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:28.709792 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:28.709802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:28.709810 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:28.709836 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:28.709884 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:28.709902 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:28.710022 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:28.710301 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 20:14:28.710446 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:14:28.711755 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:28.699851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34877 (local address 127.4.169.130:43609)
0620 20:14:28.700850 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:28.700887 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:28.702649 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:28.703383 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:28.703391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:28.703488 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:28.704097 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:28.704114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:28.705265 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:28.705271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:28.706862 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:28.706881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:28.708140 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:28.708147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:28.708466 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:28.709297 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:28.709301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:28.709353 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:28.709460 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:28.710041 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:28.710054 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:28.710062 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:14:28.710253 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 20:14:28.710644 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":323,"threads_started":1}
I20260620 20:14:28.713410 23873 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34877
I20260620 20:14:28.713764 23873 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:28.714761 23873 heartbeater.cc:511] Master 127.4.169.190:34877 requested a full tablet report, sending...
I20260620 20:14:28.716342 23556 ts_manager.cc:194] Registered new tserver with Master: fe25cd52b2e94d2fa8f4c3356586ed70 (127.4.169.130:43577)
I20260620 20:14:28.717193 23556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.130:43609
W20260620 20:14:29.037825 23876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:29.038098 23876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:29.038239 23876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:29.045848 23876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:29.045953 23876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:29.046118 23876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:14:29.055639 23876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:29.056914 23876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:29.058750 23876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:29.072371 23882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:29.072489 23881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:29.072520 23884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:29.073341 23876 server_base.cc:1034] running on GCE node
I20260620 20:14:29.074030 23876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:29.075176 23876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:29.076435 23876 hybrid_clock.cc:648] HybridClock initialized: now 1781986469076377 us; error 52 us; skew 500 ppm
I20260620 20:14:29.076808 23876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:29.079360 23876 webserver.cc:460] Webserver started at http://127.4.169.131:34991/ using document root <none> and password file <none>
I20260620 20:14:29.080271 23876 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:29.080452 23876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:29.080811 23876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:29.083983 23876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "86f11a185d0f4f08b15b8e0841e86a6d"
format_stamp: "Formatted at 2026-06-20 20:14:29 on dist-test-slave-4bwq"
server_key: "e4bb0c8857ce0ded74d1b6ac29a9400a"
server_key_iv: "0ea6b2a68fb4d63fa55e6ab36e5aa09e"
server_key_version: "encryptionkey@0"
I20260620 20:14:29.084806 23876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "86f11a185d0f4f08b15b8e0841e86a6d"
format_stamp: "Formatted at 2026-06-20 20:14:29 on dist-test-slave-4bwq"
server_key: "e4bb0c8857ce0ded74d1b6ac29a9400a"
server_key_iv: "0ea6b2a68fb4d63fa55e6ab36e5aa09e"
server_key_version: "encryptionkey@0"
I20260620 20:14:29.089979 23876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:14:29.093672 23890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:29.095233 23876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:29.095410 23876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "86f11a185d0f4f08b15b8e0841e86a6d"
format_stamp: "Formatted at 2026-06-20 20:14:29 on dist-test-slave-4bwq"
server_key: "e4bb0c8857ce0ded74d1b6ac29a9400a"
server_key_iv: "0ea6b2a68fb4d63fa55e6ab36e5aa09e"
server_key_version: "encryptionkey@0"
I20260620 20:14:29.095645 23876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:29.112599 23876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:29.113430 23876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:29.113755 23876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:29.114849 23876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:29.116294 23876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:29.116411 23876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:29.116547 23876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:29.116640 23876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:29.150363 23876 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:45545
I20260620 20:14:29.150478 24002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:45545 every 8 connection(s)
I20260620 20:14:29.151675 23876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:29.155251 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 23876
I20260620 20:14:29.155444 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:29.156018 4774 external_mini_cluster.cc:1428] Setting key ce9126a27de427c75efb9c8603836a20
I20260620 20:14:29.164384 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.154343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:42107 (local address 127.4.169.190:34877)
0620 20:14:29.154565 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.154570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.156199 (+ 1629us) server_negotiation.cc:407] Connection header received
0620 20:14:29.158319 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.158323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.158428 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.158517 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:29.159658 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.160390 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.161953 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.162353 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.162512 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.163720 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:29.163725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:29.163728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:29.163745 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:29.163773 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:29.163784 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:29.163853 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.164065 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.164150 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 20:14:29.165284 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.154882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34877 (local address 127.4.169.131:42107)
0620 20:14:29.155985 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:29.156047 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:29.158042 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:29.158783 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:29.158795 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:29.158892 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:29.159508 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.159532 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.160572 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.160576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.161824 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.161837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.162615 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.162620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.162816 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.163428 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:29.163430 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:29.163466 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:29.163553 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:29.163865 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:29.163872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:29.163878 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:14:29.164006 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 20:14:29.164269 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":303,"threads_started":1}
I20260620 20:14:29.166630 24003 heartbeater.cc:347] Connected to a master server at 127.4.169.190:34877
I20260620 20:14:29.166862 24003 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:29.167510 24003 heartbeater.cc:511] Master 127.4.169.190:34877 requested a full tablet report, sending...
I20260620 20:14:29.168781 23556 ts_manager.cc:194] Registered new tserver with Master: 86f11a185d0f4f08b15b8e0841e86a6d (127.4.169.131:45545)
I20260620 20:14:29.169595 23556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.131:42107
I20260620 20:14:29.171173 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:14:29.180759 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.175255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.4.169.190:34877)
0620 20:14:29.175441 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.175445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.175941 (+ 496us) server_negotiation.cc:407] Connection header received
0620 20:14:29.176069 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.176073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.176138 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.176218 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:29.176661 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.177337 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.178703 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.179104 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.179201 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.180184 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:29.180189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:29.180192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:29.180208 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:29.180236 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:29.180247 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:29.180353 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.180505 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.180549 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:14:29.184438 23556 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:29.191262 23556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:29.236184 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.203121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:40071 (local address 127.0.0.1:48336)
0620 20:14:29.223824 (+ 20703us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:29.223840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:29.223936 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:29.227339 (+ 3403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:29.227346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:29.227375 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:29.227852 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.227868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.229910 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.229914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.233098 (+ 3184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.233113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.235200 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.235213 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.235403 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.235462 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:14:29.235593 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 20:14:29.235780 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20563,"mutex_wait_us":638,"thread_start_us":24949,"threads_started":1}
I20260620 20:14:29.237133 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.203277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.4.169.130:43577)
0620 20:14:29.227928 (+ 24651us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.227936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.227964 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:14:29.228062 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.228070 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.228304 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.229140 (+ 836us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:29.229370 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.230655 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.233079 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.234689 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.234818 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.235059 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.236668 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.236817 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24490,"thread_start_us":24378,"threads_started":1}
I20260620 20:14:29.237149 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.203731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.4.169.129:40071)
0620 20:14:29.226792 (+ 23061us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.226798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.226825 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:14:29.226904 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.226909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.227122 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.227329 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:29.228106 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.229740 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.233302 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.234689 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.234818 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.235058 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.236606 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.236776 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22931,"thread_start_us":22832,"threads_started":1}
I20260620 20:14:29.236357 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.202759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:43577 (local address 127.0.0.1:56830)
0620 20:14:29.218978 (+ 16219us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:29.219018 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:29.219220 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:29.228520 (+ 9300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:29.228525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:29.228557 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:29.228962 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.228973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.230780 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.230788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.232878 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.232892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.235225 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.235593 (+ 368us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.235764 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.235792 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:14:29.235875 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:14:29.235995 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15955,"mutex_wait_us":3,"thread_start_us":15794,"threads_started":1}
I20260620 20:14:29.237632 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.203855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.4.169.131:45545)
0620 20:14:29.223327 (+ 19472us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.223341 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.228636 (+ 5295us) server_negotiation.cc:407] Connection header received
0620 20:14:29.228732 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.228738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.228936 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.229122 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:29.229893 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.231522 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.234167 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.235426 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.235560 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.235802 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.236899 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.237089 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19228,"thread_start_us":20130,"threads_started":1}
I20260620 20:14:29.236769 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.203487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:45545 (local address 127.0.0.1:60338)
0620 20:14:29.228342 (+ 24855us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:29.228362 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:29.228482 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:29.229392 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:29.229396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:29.229422 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:29.229706 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.229714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.231904 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.231910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.233960 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.233972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.235937 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.235944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.236121 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.236157 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:14:29.236262 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:14:29.236402 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24733,"mutex_wait_us":1,"thread_start_us":20055,"threads_started":1}
I20260620 20:14:29.241715 23938 tablet_service.cc:1467] Processing CreateTablet for tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e (DEFAULT_TABLE table=test-table [id=3a31e7a0320c4b2180e3a259b03c2b7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:29.241719 23677 tablet_service.cc:1467] Processing CreateTablet for tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e (DEFAULT_TABLE table=test-table [id=3a31e7a0320c4b2180e3a259b03c2b7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:29.241719 23808 tablet_service.cc:1467] Processing CreateTablet for tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e (DEFAULT_TABLE table=test-table [id=3a31e7a0320c4b2180e3a259b03c2b7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:29.243060 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:29.243098 23677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:29.243114 23808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:29.257678 24021 tablet_bootstrap.cc:492] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: Bootstrap starting.
I20260620 20:14:29.261857 24023 tablet_bootstrap.cc:492] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: Bootstrap starting.
I20260620 20:14:29.262449 24021 tablet_bootstrap.cc:654] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:29.263088 24022 tablet_bootstrap.cc:492] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: Bootstrap starting.
I20260620 20:14:29.263216 23742 heartbeater.cc:503] Master 127.4.169.190:34877 was elected leader, sending a full tablet report...
I20260620 20:14:29.263885 24021 log.cc:824] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:29.265797 24023 tablet_bootstrap.cc:654] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:29.266520 24022 tablet_bootstrap.cc:654] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:29.267055 24023 log.cc:824] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:29.267679 24022 log.cc:824] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:29.273244 24022 tablet_bootstrap.cc:492] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: No bootstrap required, opened a new log
I20260620 20:14:29.273511 24022 ts_tablet_manager.cc:1397] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 20:14:29.273795 24023 tablet_bootstrap.cc:492] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: No bootstrap required, opened a new log
I20260620 20:14:29.273857 24021 tablet_bootstrap.cc:492] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: No bootstrap required, opened a new log
I20260620 20:14:29.274097 24023 ts_tablet_manager.cc:1397] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.004s
I20260620 20:14:29.274260 24021 ts_tablet_manager.cc:1397] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260620 20:14:29.280827 24023 raft_consensus.cc:348] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } }
I20260620 20:14:29.281263 24023 raft_consensus.cc:374] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:29.281394 24023 raft_consensus.cc:729] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8aef393522453698157a20594a9641, State: Initialized, Role: FOLLOWER
I20260620 20:14:29.282018 24023 consensus_queue.cc:260] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } }
I20260620 20:14:29.283982 24023 ts_tablet_manager.cc:1428] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260620 20:14:29.284090 24022 raft_consensus.cc:348] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } }
I20260620 20:14:29.284597 24022 raft_consensus.cc:374] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:29.284740 24022 raft_consensus.cc:729] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe25cd52b2e94d2fa8f4c3356586ed70, State: Initialized, Role: FOLLOWER
I20260620 20:14:29.284741 24021 raft_consensus.cc:348] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } }
I20260620 20:14:29.285212 24021 raft_consensus.cc:374] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:29.285343 24021 raft_consensus.cc:729] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f11a185d0f4f08b15b8e0841e86a6d, State: Initialized, Role: FOLLOWER
I20260620 20:14:29.285490 24022 consensus_queue.cc:260] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } }
I20260620 20:14:29.286044 24021 consensus_queue.cc:260] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } }
I20260620 20:14:29.287320 24022 ts_tablet_manager.cc:1428] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 20:14:29.287415 24003 heartbeater.cc:503] Master 127.4.169.190:34877 was elected leader, sending a full tablet report...
I20260620 20:14:29.287845 23873 heartbeater.cc:503] Master 127.4.169.190:34877 was elected leader, sending a full tablet report...
I20260620 20:14:29.287956 24021 ts_tablet_manager.cc:1428] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260620 20:14:29.403395 24004 tablet.cc:2369] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:29.448643 23874 tablet.cc:2369] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:29.489571 23743 tablet.cc:2369] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:29.502763 24029 raft_consensus.cc:482] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:29.503078 24029 raft_consensus.cc:504] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } }
I20260620 20:14:29.504798 24029 leader_election.cc:288] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe25cd52b2e94d2fa8f4c3356586ed70 (127.4.169.130:43577), 1c8aef393522453698157a20594a9641 (127.4.169.129:40071)
I20260620 20:14:29.511613 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.505133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:44457 (local address 127.4.169.130:43577)
0620 20:14:29.505331 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.505337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.505553 (+ 216us) server_negotiation.cc:407] Connection header received
0620 20:14:29.505731 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.505735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.505808 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.505885 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:29.506335 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.507321 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.509674 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.510782 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.510929 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.511000 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.511231 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.511371 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:14:29.511642 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.505052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:43577 (local address 127.4.169.131:44457)
0620 20:14:29.505384 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:29.505405 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:29.505560 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:29.505956 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:29.505960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:29.505983 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:29.506169 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.506177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.507454 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.507457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.509509 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.509524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.510927 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.510931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.511067 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.511108 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:14:29.511178 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:14:29.511277 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":4}
I20260620 20:14:29.512233 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.505231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:40789 (local address 127.4.169.129:40071)
0620 20:14:29.505465 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.505474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.505948 (+ 474us) server_negotiation.cc:407] Connection header received
0620 20:14:29.506094 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.506098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.506164 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.506234 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:29.506812 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.507719 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.510014 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.511264 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.511408 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.511469 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.511841 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.511982 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:14:29.512755 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.505059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:40071 (local address 127.4.169.131:40789)
0620 20:14:29.505814 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:29.505830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:29.505958 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:29.506281 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:29.506285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:29.506310 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:29.506658 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.506666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.507862 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.507867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.509851 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:29.509861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.511442 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.511448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:29.511637 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.511680 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 20:14:29.512428 (+ 748us) client_negotiation.cc:240] Negotiation successful
0620 20:14:29.512528 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":171,"thread_start_us":293,"threads_started":1}
I20260620 20:14:29.512697 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c4fbcbab204d2eaff4cbf7bbfd7a5e" candidate_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" is_pre_election: true
I20260620 20:14:29.513283 23828 raft_consensus.cc:2454] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86f11a185d0f4f08b15b8e0841e86a6d in term 0.
I20260620 20:14:29.513772 23697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c4fbcbab204d2eaff4cbf7bbfd7a5e" candidate_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8aef393522453698157a20594a9641" is_pre_election: true
I20260620 20:14:29.514019 23891 leader_election.cc:302] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86f11a185d0f4f08b15b8e0841e86a6d, fe25cd52b2e94d2fa8f4c3356586ed70; no voters:
I20260620 20:14:29.514320 23697 raft_consensus.cc:2454] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86f11a185d0f4f08b15b8e0841e86a6d in term 0.
I20260620 20:14:29.514571 24029 raft_consensus.cc:2781] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:29.514739 24029 raft_consensus.cc:482] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:29.514869 24029 raft_consensus.cc:3037] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:29.517139 24029 raft_consensus.cc:504] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } }
I20260620 20:14:29.517735 24029 leader_election.cc:288] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [CANDIDATE]: Term 1 election: Requested vote from peers fe25cd52b2e94d2fa8f4c3356586ed70 (127.4.169.130:43577), 1c8aef393522453698157a20594a9641 (127.4.169.129:40071)
I20260620 20:14:29.518117 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c4fbcbab204d2eaff4cbf7bbfd7a5e" candidate_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70"
I20260620 20:14:29.518138 23697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c4fbcbab204d2eaff4cbf7bbfd7a5e" candidate_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8aef393522453698157a20594a9641"
I20260620 20:14:29.518332 23828 raft_consensus.cc:3037] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:29.518337 23697 raft_consensus.cc:3037] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:29.520640 23828 raft_consensus.cc:2454] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86f11a185d0f4f08b15b8e0841e86a6d in term 1.
I20260620 20:14:29.520640 23697 raft_consensus.cc:2454] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86f11a185d0f4f08b15b8e0841e86a6d in term 1.
I20260620 20:14:29.521052 23891 leader_election.cc:302] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86f11a185d0f4f08b15b8e0841e86a6d, fe25cd52b2e94d2fa8f4c3356586ed70; no voters:
I20260620 20:14:29.521327 24029 raft_consensus.cc:2781] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:29.521750 24029 raft_consensus.cc:686] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 1 LEADER]: Becoming Leader. State: Replica: 86f11a185d0f4f08b15b8e0841e86a6d, State: Running, Role: LEADER
I20260620 20:14:29.522260 24029 consensus_queue.cc:237] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } }
I20260620 20:14:29.527341 23556 catalog_manager.cc:5495] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 86f11a185d0f4f08b15b8e0841e86a6d (127.4.169.131). New cstate: current_term: 1 leader_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86f11a185d0f4f08b15b8e0841e86a6d" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 45545 } health_report { overall_health: HEALTHY } } }
I20260620 20:14:29.613364 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.607881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.4.169.131:45545)
0620 20:14:29.608079 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.608084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.608106 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:14:29.608205 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.608210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.608309 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.608457 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:29.608862 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.609613 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.611133 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.611554 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.611688 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.612278 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:29.612778 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:29.612963 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.613100 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.613145 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:14:29.621268 23828 raft_consensus.cc:1261] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [term 1 FOLLOWER]: Refusing update from remote peer 86f11a185d0f4f08b15b8e0841e86a6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:29.621268 23697 raft_consensus.cc:1261] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [term 1 FOLLOWER]: Refusing update from remote peer 86f11a185d0f4f08b15b8e0841e86a6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:29.622259 24029 consensus_queue.cc:1035] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8aef393522453698157a20594a9641" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 40071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:29.622611 24031 consensus_queue.cc:1035] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe25cd52b2e94d2fa8f4c3356586ed70" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:29.632383 24035 mvcc.cc:205] Tried to move back new op lower bound from 7299016579559227392 to 7299016579167608832. Current Snapshot: MvccSnapshot[applied={T|T < 7299016579559227392}]
I20260620 20:14:29.632464 24034 mvcc.cc:205] Tried to move back new op lower bound from 7299016579559227392 to 7299016579167608832. Current Snapshot: MvccSnapshot[applied={T|T < 7299016579559227392}]
I20260620 20:14:29.647727 23556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:29.648001 23556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:29.651769 23556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cf46d6a87e24aaeaf809d69a0a780ca: Sending DeleteTablet for 3 replicas of tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e
I20260620 20:14:29.652985 23808 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:29 UTC) from {username='slave'} at 127.0.0.1:56830
I20260620 20:14:29.652985 23677 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:29 UTC) from {username='slave'} at 127.0.0.1:48336
I20260620 20:14:29.653885 24045 tablet_replica.cc:330] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: stopping tablet replica
I20260620 20:14:29.654366 23938 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:29 UTC) from {username='slave'} at 127.0.0.1:60338
I20260620 20:14:29.654651 24044 tablet_replica.cc:330] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: stopping tablet replica
I20260620 20:14:29.654915 24045 raft_consensus.cc:2229] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:29.655098 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 23615
I20260620 20:14:29.655066 24044 raft_consensus.cc:2229] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:29.655546 24044 raft_consensus.cc:2258] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:29.655594 24045 raft_consensus.cc:2258] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:29.657855 24044 ts_tablet_manager.cc:1905] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:29.657999 24045 ts_tablet_manager.cc:1905] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:29.666136 24045 ts_tablet_manager.cc:1918] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:29.666489 24045 log.cc:1196] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/36c4fbcbab204d2eaff4cbf7bbfd7a5e
I20260620 20:14:29.667012 24044 ts_tablet_manager.cc:1918] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:29.667264 24045 ts_tablet_manager.cc:1939] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 1c8aef393522453698157a20594a9641: Deleting consensus metadata
I20260620 20:14:29.667279 24044 log.cc:1196] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/36c4fbcbab204d2eaff4cbf7bbfd7a5e
I20260620 20:14:29.667847 24044 ts_tablet_manager.cc:1939] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P fe25cd52b2e94d2fa8f4c3356586ed70: Deleting consensus metadata
I20260620 20:14:29.668062 24046 tablet_replica.cc:330] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: stopping tablet replica
I20260620 20:14:29.668503 24046 raft_consensus.cc:2229] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:29.669173 24046 raft_consensus.cc:2258] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:29.669389 23541 catalog_manager.cc:4841] TS 1c8aef393522453698157a20594a9641 (127.4.169.129:40071): tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e (table test-table [id=3a31e7a0320c4b2180e3a259b03c2b7e]) successfully deleted
I20260620 20:14:29.670504 23540 catalog_manager.cc:4841] TS fe25cd52b2e94d2fa8f4c3356586ed70 (127.4.169.130:43577): tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e (table test-table [id=3a31e7a0320c4b2180e3a259b03c2b7e]) successfully deleted
I20260620 20:14:29.672323 24046 ts_tablet_manager.cc:1905] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:29.674335 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.658511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.4.169.129:40071)
0620 20:14:29.658686 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.658692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.667146 (+ 8454us) server_negotiation.cc:407] Connection header received
0620 20:14:29.667428 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.667434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.667561 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.667675 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:29.668445 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.669508 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.671312 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.671739 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.671879 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.672991 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:29.673006 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:29.673018 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:29.673554 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:29.673594 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:29.673613 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:29.673733 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.673864 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.673979 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:14:29.675014 23737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:29.679838 24046 ts_tablet_manager.cc:1918] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:29.680003 24046 log.cc:1196] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/36c4fbcbab204d2eaff4cbf7bbfd7a5e
I20260620 20:14:29.680567 24046 ts_tablet_manager.cc:1939] T 36c4fbcbab204d2eaff4cbf7bbfd7a5e P 86f11a185d0f4f08b15b8e0841e86a6d: Deleting consensus metadata
I20260620 20:14:29.682361 23543 catalog_manager.cc:4841] TS 86f11a185d0f4f08b15b8e0841e86a6d (127.4.169.131:45545): tablet 36c4fbcbab204d2eaff4cbf7bbfd7a5e (table test-table [id=3a31e7a0320c4b2180e3a259b03c2b7e]) successfully deleted
I20260620 20:14:29.780393 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 23615
I20260620 20:14:29.798098 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 23746
I20260620 20:14:29.806057 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.799006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.4.169.130:43577)
0620 20:14:29.799210 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.799216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.799238 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:14:29.799308 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.799315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.799424 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.799535 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:29.799976 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.801118 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.802548 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.803206 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.803337 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.804391 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:29.804402 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:29.804420 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:29.805207 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:29.805267 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:29.805291 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:29.805408 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.805572 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.805682 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:14:29.806790 23868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:29.895642 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 23746
I20260620 20:14:29.913028 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 23876
I20260620 20:14:29.921415 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:29.914185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.4.169.131:45545)
0620 20:14:29.914418 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:29.914429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:29.914452 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:14:29.914531 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:29.914537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:29.914653 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:29.914791 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:29.915199 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.916451 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.917857 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:29.918579 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:29.918727 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:29.919740 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:29.919750 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:29.919768 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:29.920560 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:29.920619 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:29.920644 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:29.920775 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:29.920919 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 20:14:29.921019 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:14:29.922166 23998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:30.042439 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 23876
I20260620 20:14:30.061863 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 23525
I20260620 20:14:30.063043 23586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:30.170194 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 23525
2026-06-20T20:14:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3005 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:30 dist-test-slave-4bwq krb5kdc[24054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:30 dist-test-slave-4bwq krb5kdc[24054](info): set up 2 sockets
Jun 20 20:14:30 dist-test-slave-4bwq krb5kdc[24054](info): commencing operation
krb5kdc: starting...
W20260620 20:14:31.274972 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:31 dist-test-slave-4bwq krb5kdc[24054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:14:31.541910 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:38417
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:31.882130 24067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:31.882448 24067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:31.882540 24067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:31.889930 24067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:31.890053 24067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:31.890105 24067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:31.890163 24067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:31.890208 24067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:31.903978 24067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:31.905283 24067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:31.906610 24067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:31.918900 24072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:31.918962 24075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:31.918937 24073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:31.919801 24067 server_base.cc:1034] running on GCE node
I20260620 20:14:31.920620 24067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:31.922170 24067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:31.923401 24067 hybrid_clock.cc:648] HybridClock initialized: now 1781986471923353 us; error 64 us; skew 500 ppm
I20260620 20:14:31.923767 24067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:31 dist-test-slave-4bwq krb5kdc[24054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:31.932585 24067 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:14:31.934110 24067 webserver.cc:460] Webserver started at http://127.4.169.190:43693/ using document root <none> and password file <none>
I20260620 20:14:31.934939 24067 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:31.935086 24067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:31.935390 24067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:31.938349 24067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "bffae9b1ee7748788b9ae95f70e7a110"
format_stamp: "Formatted at 2026-06-20 20:14:31 on dist-test-slave-4bwq"
server_key: "bcbd26c7faa3a56259905899f485a964"
server_key_iv: "f3df6628860cee42769146709240176c"
server_key_version: "encryptionkey@0"
I20260620 20:14:31.939153 24067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "bffae9b1ee7748788b9ae95f70e7a110"
format_stamp: "Formatted at 2026-06-20 20:14:31 on dist-test-slave-4bwq"
server_key: "bcbd26c7faa3a56259905899f485a964"
server_key_iv: "f3df6628860cee42769146709240176c"
server_key_version: "encryptionkey@0"
I20260620 20:14:31.944173 24067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:14:31.947508 24082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:31.948817 24067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:14:31.948994 24067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "bffae9b1ee7748788b9ae95f70e7a110"
format_stamp: "Formatted at 2026-06-20 20:14:31 on dist-test-slave-4bwq"
server_key: "bcbd26c7faa3a56259905899f485a964"
server_key_iv: "f3df6628860cee42769146709240176c"
server_key_version: "encryptionkey@0"
I20260620 20:14:31.949184 24067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:32.008536 24067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:32.013231 24067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:32.013543 24067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:32.032861 24067 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:38417
I20260620 20:14:32.032886 24133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:38417 every 8 connection(s)
I20260620 20:14:32.034317 24067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:32.037732 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 24067
I20260620 20:14:32.037925 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:32.038203 24134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:32.038578 4774 external_mini_cluster.cc:1428] Setting key 96970cedd0898f4873ba72b3deaf834e
I20260620 20:14:32.052780 24134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110: Bootstrap starting.
Jun 20 20:14:32 dist-test-slave-4bwq krb5kdc[24054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:32.056430 24134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:32.057869 24134 log.cc:824] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:32.061283 24134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110: No bootstrap required, opened a new log
I20260620 20:14:32.063748 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:32.040400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.4.169.190:38417)
0620 20:14:32.041232 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:32.041248 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:32.041300 (+ 52us) server_negotiation.cc:407] Connection header received
0620 20:14:32.043427 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:32.043455 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:32.043963 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:32.044393 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:32.045726 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:32.047834 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:32.049166 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:32.050251 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:32.050456 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:32.054856 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:32.054869 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:32.054889 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:32.054938 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:32.058998 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:32.059789 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:32.059797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:32.059805 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:32.060011 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:32.060429 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:32.060433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:32.060438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:32.061134 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:32.061444 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:32.061906 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 20:14:32.062364 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":327,"threads_started":1}
I20260620 20:14:32.070318 24134 raft_consensus.cc:348] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffae9b1ee7748788b9ae95f70e7a110" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38417 } }
I20260620 20:14:32.070786 24134 raft_consensus.cc:374] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:32.070865 24134 raft_consensus.cc:729] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffae9b1ee7748788b9ae95f70e7a110, State: Initialized, Role: FOLLOWER
I20260620 20:14:32.071470 24134 consensus_queue.cc:260] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffae9b1ee7748788b9ae95f70e7a110" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38417 } }
I20260620 20:14:32.071800 24134 raft_consensus.cc:388] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:32.071995 24134 raft_consensus.cc:482] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:32.072190 24134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:32.074568 24134 raft_consensus.cc:504] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffae9b1ee7748788b9ae95f70e7a110" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38417 } }
I20260620 20:14:32.075130 24134 leader_election.cc:302] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bffae9b1ee7748788b9ae95f70e7a110; no voters:
I20260620 20:14:32.075733 24134 leader_election.cc:288] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:32.075873 24139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:32.076618 24139 raft_consensus.cc:686] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [term 1 LEADER]: Becoming Leader. State: Replica: bffae9b1ee7748788b9ae95f70e7a110, State: Running, Role: LEADER
I20260620 20:14:32.077191 24139 consensus_queue.cc:237] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffae9b1ee7748788b9ae95f70e7a110" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38417 } }
I20260620 20:14:32.078066 24134 sys_catalog.cc:564] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:32.080041 24141 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bffae9b1ee7748788b9ae95f70e7a110. Latest consensus state: current_term: 1 leader_uuid: "bffae9b1ee7748788b9ae95f70e7a110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffae9b1ee7748788b9ae95f70e7a110" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38417 } } }
I20260620 20:14:32.080439 24141 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:32.082667 24140 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bffae9b1ee7748788b9ae95f70e7a110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffae9b1ee7748788b9ae95f70e7a110" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38417 } } }
I20260620 20:14:32.083555 24146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:32.083601 24140 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:32.088762 24146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:32.097504 24146 catalog_manager.cc:1347] Generated new cluster ID: ca1416f764754916b17ae07407be9de0
I20260620 20:14:32.097663 24146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:32.108017 24146 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:32.110036 24146 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:32.119099 24146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110: Generated new TSK 0
I20260620 20:14:32.120250 24146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.169.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
Entry for principal kudu/127.4.169.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab.
I20260620 20:14:32.189563 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38417
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:32.536594 24160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:32.536926 24160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:32.537113 24160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:32.544327 24160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:32.544481 24160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:32.544673 24160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:14:32.554555 24160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.129.keytab
--principal=kudu/127.4.169.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:32.555899 24160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:32.557279 24160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:32.571028 24165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:32.571070 24166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:32.571125 24168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:32.571827 24160 server_base.cc:1034] running on GCE node
I20260620 20:14:32.572510 24160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:32.573501 24160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:32.574699 24160 hybrid_clock.cc:648] HybridClock initialized: now 1781986472574652 us; error 36 us; skew 500 ppm
I20260620 20:14:32.575031 24160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:32 dist-test-slave-4bwq krb5kdc[24054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986472, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:32.578676 24160 init.cc:377] Logged in from keytab as kudu/127.4.169.129@KRBTEST.COM (short username kudu)
I20260620 20:14:32.580296 24160 webserver.cc:460] Webserver started at http://127.4.169.129:42787/ using document root <none> and password file <none>
I20260620 20:14:32.581030 24160 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:32.581182 24160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:32.581580 24160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:32.584715 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "3a1c0aafbb254637802b002b22c4a404"
format_stamp: "Formatted at 2026-06-20 20:14:32 on dist-test-slave-4bwq"
server_key: "41796c1e9f0c591aa9bc1230c24d050f"
server_key_iv: "e79df3422fda0e2ac19c6e383183ba1c"
server_key_version: "encryptionkey@0"
I20260620 20:14:32.585520 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "3a1c0aafbb254637802b002b22c4a404"
format_stamp: "Formatted at 2026-06-20 20:14:32 on dist-test-slave-4bwq"
server_key: "41796c1e9f0c591aa9bc1230c24d050f"
server_key_iv: "e79df3422fda0e2ac19c6e383183ba1c"
server_key_version: "encryptionkey@0"
I20260620 20:14:32.590922 24160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:14:32.594496 24175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:32.595852 24160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:32.596019 24160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "3a1c0aafbb254637802b002b22c4a404"
format_stamp: "Formatted at 2026-06-20 20:14:32 on dist-test-slave-4bwq"
server_key: "41796c1e9f0c591aa9bc1230c24d050f"
server_key_iv: "e79df3422fda0e2ac19c6e383183ba1c"
server_key_version: "encryptionkey@0"
I20260620 20:14:32.596241 24160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:32.611690 24160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:32.616336 24160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:32.616685 24160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:32.618155 24160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:32.620460 24160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:32.620534 24160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:32.620677 24160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:32.620743 24160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:32.654621 24160 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:44541
I20260620 20:14:32.654663 24287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:44541 every 8 connection(s)
I20260620 20:14:32.655954 24160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:32.659238 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 24160
I20260620 20:14:32.659426 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:32.659904 4774 external_mini_cluster.cc:1428] Setting key 6b534634b52673308396381ae8672f25
Jun 20 20:14:32 dist-test-slave-4bwq krb5kdc[24054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986472, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.129@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:32.680480 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:32.658555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:58327 (local address 127.4.169.190:38417)
0620 20:14:32.659128 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:32.659134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:32.660259 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 20:14:32.662868 (+ 2609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:32.662872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:32.662950 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:32.663053 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:32.665308 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:32.666288 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:32.668902 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:32.669575 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:32.669717 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:32.673872 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:32.673877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:32.673881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:32.673916 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:32.676561 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:32.677323 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:32.677333 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:32.677335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:32.677456 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:32.677951 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:32.677959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:32.677965 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:32.678494 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:32.678654 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:32.679700 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0620 20:14:32.679974 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":239,"threads_started":1}
I20260620 20:14:32.681156 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:32.659198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38417 (local address 127.4.169.129:58327)
0620 20:14:32.660063 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:32.660129 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:32.662542 (+ 2413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:32.663355 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:32.663370 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:32.664211 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:32.665117 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:32.665146 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:32.666495 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:32.666503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:32.668724 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:32.668744 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:32.669837 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:32.669846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:32.670206 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:32.671025 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:32.671062 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:32.673631 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:32.676706 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:32.676716 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:32.676735 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:32.677193 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:32.677572 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:32.677577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:32.677580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:32.677787 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:32.678624 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:32.678630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:32.679147 (+ 517us) client_negotiation.cc:769] Sending connection context
0620 20:14:32.679576 (+ 429us) client_negotiation.cc:240] Negotiation successful
0620 20:14:32.680012 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":385,"threads_started":1}
I20260620 20:14:32.682909 24288 heartbeater.cc:347] Connected to a master server at 127.4.169.190:38417
I20260620 20:14:32.683298 24288 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:32.684335 24288 heartbeater.cc:511] Master 127.4.169.190:38417 requested a full tablet report, sending...
I20260620 20:14:32.687021 24099 ts_manager.cc:194] Registered new tserver with Master: 3a1c0aafbb254637802b002b22c4a404 (127.4.169.129:44541)
WARNING: no policy specified for kudu/127.4.169.130@KRBTEST.COM; defaulting to no policy
I20260620 20:14:32.688592 24099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.129@KRBTEST.COM'} at 127.4.169.129:58327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
Entry for principal kudu/127.4.169.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab.
I20260620 20:14:32.721235 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38417
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:33.068967 24294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:33.069311 24294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:33.069494 24294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:33.077642 24294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:33.077821 24294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:33.078002 24294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:14:33.088649 24294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.130.keytab
--principal=kudu/127.4.169.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:33.090121 24294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:33.091770 24294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:33.105921 24300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:33.105989 24299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:33.105952 24302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:33.106712 24294 server_base.cc:1034] running on GCE node
I20260620 20:14:33.107396 24294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:33.108433 24294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:33.109647 24294 hybrid_clock.cc:648] HybridClock initialized: now 1781986473109600 us; error 59 us; skew 500 ppm
I20260620 20:14:33.109983 24294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:33 dist-test-slave-4bwq krb5kdc[24054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986473, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:33.113480 24294 init.cc:377] Logged in from keytab as kudu/127.4.169.130@KRBTEST.COM (short username kudu)
I20260620 20:14:33.115278 24294 webserver.cc:460] Webserver started at http://127.4.169.130:36277/ using document root <none> and password file <none>
I20260620 20:14:33.116032 24294 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:33.116165 24294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:33.116500 24294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:33.119692 24294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "41975959a1374a72bc088c8a3eb75ea2"
format_stamp: "Formatted at 2026-06-20 20:14:33 on dist-test-slave-4bwq"
server_key: "a717e26f652f6c3190b6449b76512733"
server_key_iv: "232bb26a3bc72481b5123abb3240a0be"
server_key_version: "encryptionkey@0"
I20260620 20:14:33.120633 24294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "41975959a1374a72bc088c8a3eb75ea2"
format_stamp: "Formatted at 2026-06-20 20:14:33 on dist-test-slave-4bwq"
server_key: "a717e26f652f6c3190b6449b76512733"
server_key_iv: "232bb26a3bc72481b5123abb3240a0be"
server_key_version: "encryptionkey@0"
I20260620 20:14:33.126343 24294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:14:33.129832 24309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:33.131196 24294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:33.131373 24294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "41975959a1374a72bc088c8a3eb75ea2"
format_stamp: "Formatted at 2026-06-20 20:14:33 on dist-test-slave-4bwq"
server_key: "a717e26f652f6c3190b6449b76512733"
server_key_iv: "232bb26a3bc72481b5123abb3240a0be"
server_key_version: "encryptionkey@0"
I20260620 20:14:33.131587 24294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:33.149674 24294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:33.154124 24294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:33.154458 24294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:33.155509 24294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:33.156863 24294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:33.156978 24294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:33.157137 24294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:33.157210 24294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:33.192374 24294 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:40031
I20260620 20:14:33.192420 24421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:40031 every 8 connection(s)
I20260620 20:14:33.193739 24294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:33.195950 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 24294
I20260620 20:14:33.196162 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:33.196861 4774 external_mini_cluster.cc:1428] Setting key 8d3dc8454f05461bba9c6eb15c7b0d19
Jun 20 20:14:33 dist-test-slave-4bwq krb5kdc[24054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986473, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.130@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:33.217680 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.196543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:39727 (local address 127.4.169.190:38417)
0620 20:14:33.198141 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:33.198147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:33.199111 (+ 964us) server_negotiation.cc:407] Connection header received
0620 20:14:33.201334 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:33.201344 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:33.201443 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:33.201561 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:33.203565 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.204734 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.206348 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.207125 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.207270 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.211476 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:33.211482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:33.211485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:33.211520 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:33.214313 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:33.215163 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:33.215170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:33.215174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:33.215311 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:33.215769 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:33.215774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:33.215777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:33.216074 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:33.216236 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:33.217022 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 20:14:33.217263 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1470,"thread_start_us":1357,"threads_started":1}
I20260620 20:14:33.218794 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.197257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38417 (local address 127.4.169.130:39727)
0620 20:14:33.198917 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:33.198966 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:33.200903 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:33.201860 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:33.201870 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:33.202743 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:33.203354 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.203373 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.204898 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.204902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.206188 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.206201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.207347 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.207353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.207592 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.208367 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:33.208407 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:33.211226 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:33.214513 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:33.214523 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:33.214541 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:33.215008 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:33.215430 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:33.215436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:33.215440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:33.215647 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:33.216207 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:33.216214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:33.216600 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 20:14:33.216920 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 20:14:33.217517 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"thread_start_us":1102,"threads_started":1}
I20260620 20:14:33.220463 24422 heartbeater.cc:347] Connected to a master server at 127.4.169.190:38417
I20260620 20:14:33.220827 24422 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:33.221590 24422 heartbeater.cc:511] Master 127.4.169.190:38417 requested a full tablet report, sending...
I20260620 20:14:33.223178 24099 ts_manager.cc:194] Registered new tserver with Master: 41975959a1374a72bc088c8a3eb75ea2 (127.4.169.130:40031)
WARNING: no policy specified for kudu/127.4.169.131@KRBTEST.COM; defaulting to no policy
I20260620 20:14:33.224025 24099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.130@KRBTEST.COM'} at 127.4.169.130:39727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
Entry for principal kudu/127.4.169.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab.
I20260620 20:14:33.255436 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38417
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:33.599485 24428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:33.599879 24428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:33.600055 24428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:33.608068 24428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:33.608229 24428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:33.608400 24428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:14:33.618571 24428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.131.keytab
--principal=kudu/127.4.169.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:38417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:33.620028 24428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:33.622097 24428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:33.635994 24433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:33.636005 24436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:33.636026 24434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:33.636868 24428 server_base.cc:1034] running on GCE node
I20260620 20:14:33.637460 24428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:33.638854 24428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:33.640133 24428 hybrid_clock.cc:648] HybridClock initialized: now 1781986473640045 us; error 72 us; skew 500 ppm
I20260620 20:14:33.640473 24428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:33 dist-test-slave-4bwq krb5kdc[24054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986473, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:33.643821 24428 init.cc:377] Logged in from keytab as kudu/127.4.169.131@KRBTEST.COM (short username kudu)
I20260620 20:14:33.645304 24428 webserver.cc:460] Webserver started at http://127.4.169.131:37675/ using document root <none> and password file <none>
I20260620 20:14:33.646037 24428 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:33.646195 24428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:33.646600 24428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:33.649610 24428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "c6bc55f1e4a543878ac41816e44ea10c"
format_stamp: "Formatted at 2026-06-20 20:14:33 on dist-test-slave-4bwq"
server_key: "b5fb2192b643c8a5290919e3d9748630"
server_key_iv: "f4b34ad4d51deab89237c302d39ffeab"
server_key_version: "encryptionkey@0"
I20260620 20:14:33.650465 24428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "c6bc55f1e4a543878ac41816e44ea10c"
format_stamp: "Formatted at 2026-06-20 20:14:33 on dist-test-slave-4bwq"
server_key: "b5fb2192b643c8a5290919e3d9748630"
server_key_iv: "f4b34ad4d51deab89237c302d39ffeab"
server_key_version: "encryptionkey@0"
I20260620 20:14:33.655820 24428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:14:33.659521 24443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:33.660854 24428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:14:33.661041 24428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "c6bc55f1e4a543878ac41816e44ea10c"
format_stamp: "Formatted at 2026-06-20 20:14:33 on dist-test-slave-4bwq"
server_key: "b5fb2192b643c8a5290919e3d9748630"
server_key_iv: "f4b34ad4d51deab89237c302d39ffeab"
server_key_version: "encryptionkey@0"
I20260620 20:14:33.661262 24428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:33.691790 24428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:33.692265 24288 heartbeater.cc:503] Master 127.4.169.190:38417 was elected leader, sending a full tablet report...
I20260620 20:14:33.696830 24428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:33.697199 24428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:33.698230 24428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:33.699733 24428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:33.699834 24428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:33.700001 24428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:33.700083 24428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:33.736295 24428 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:46259
I20260620 20:14:33.736341 24555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:46259 every 8 connection(s)
I20260620 20:14:33.737617 24428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:33.740772 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 24428
I20260620 20:14:33.740979 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:33.741420 4774 external_mini_cluster.cc:1428] Setting key 9fd10bb89c69e28f032333c9f35eac1a
Jun 20 20:14:33 dist-test-slave-4bwq krb5kdc[24054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986473, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.131@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:14:33.758646 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.740391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:38525 (local address 127.4.169.190:38417)
0620 20:14:33.740897 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:33.740903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:33.741928 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 20:14:33.744087 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:33.744091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:33.744182 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:33.744332 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:33.745890 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.746818 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.748351 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.748773 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.748876 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.752242 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:33.752247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:33.752250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:33.752283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:33.755291 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:33.756153 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:33.756158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:33.756162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:33.756318 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:33.756780 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:33.756786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:33.756790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:33.757116 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:33.757248 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:33.758050 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 20:14:33.758239 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":247,"threads_started":1}
I20260620 20:14:33.759522 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.740916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:38417 (local address 127.4.169.131:38525)
0620 20:14:33.741756 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:33.741804 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:33.743780 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:33.744595 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:33.744606 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:33.745118 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:33.745726 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.745744 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.746963 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.746967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.748205 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.748216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.748955 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.748959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.749192 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.749807 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:33.749828 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:33.752047 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:33.755418 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:33.755424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:33.755435 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:33.756024 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:33.756427 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:33.756432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:33.756436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:33.756654 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:33.757266 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:33.757272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:33.757649 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 20:14:33.757893 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 20:14:33.758300 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":373,"threads_started":1}
I20260620 20:14:33.760900 24556 heartbeater.cc:347] Connected to a master server at 127.4.169.190:38417
I20260620 20:14:33.761190 24556 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:33.761878 24556 heartbeater.cc:511] Master 127.4.169.190:38417 requested a full tablet report, sending...
I20260620 20:14:33.763345 24099 ts_manager.cc:194] Registered new tserver with Master: c6bc55f1e4a543878ac41816e44ea10c (127.4.169.131:46259)
I20260620 20:14:33.764217 24099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.169.131@KRBTEST.COM'} at 127.4.169.131:38525
I20260620 20:14:33.769171 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:14:33.786657 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.773412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.4.169.190:38417)
0620 20:14:33.773587 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:33.773592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:33.773962 (+ 370us) server_negotiation.cc:407] Connection header received
0620 20:14:33.774110 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:33.774114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:33.774178 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:33.774260 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:33.775751 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.776496 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.777822 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.778256 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.778352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.780944 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:33.780950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:33.780953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:33.780987 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:33.783862 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:33.784637 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:33.784641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:33.784644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:33.784776 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:33.785218 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:33.785222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:33.785224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:33.785567 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:33.785773 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:33.786145 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 20:14:33.786329 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:14:33.790896 24099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:33.797979 24099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:33.824308 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.811156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:40031 (local address 127.0.0.1:55366)
0620 20:14:33.812198 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:33.812268 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:33.812489 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:33.814065 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:33.814077 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:33.814110 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:33.814564 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.814578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.817353 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.817363 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.820522 (+ 3159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.820549 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.823162 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.823168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.823405 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.823473 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 20:14:33.823600 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 20:14:33.823851 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":573,"threads_started":1}
I20260620 20:14:33.825768 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.811443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.4.169.130:40031)
0620 20:14:33.813206 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:33.813215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:33.813248 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:14:33.813366 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:33.813373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:33.813646 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:33.814809 (+ 1163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:33.815064 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.817141 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.820871 (+ 3730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.822901 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.824882 (+ 1981us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.825144 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:33.825234 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 20:14:33.825415 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":1487,"threads_started":1}
I20260620 20:14:33.829921 24357 tablet_service.cc:1467] Processing CreateTablet for tablet 05e71a3f0cd74e6997e7d65a0002cdba (DEFAULT_TABLE table=test-table [id=5b817ff0fd26422ea585ab92547e27e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:33.831331 24357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e71a3f0cd74e6997e7d65a0002cdba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:33.832687 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.812660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:46259 (local address 127.0.0.1:57236)
0620 20:14:33.824619 (+ 11959us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:33.824640 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:33.824742 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:33.825903 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:33.825908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:33.825935 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:33.826139 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.826152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.828031 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.828037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.829908 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.829920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.831809 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.831815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.832004 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.832037 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:14:33.832147 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:14:33.832301 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11840,"mutex_wait_us":17,"thread_start_us":11177,"threads_started":1}
I20260620 20:14:33.832739 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.813497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.4.169.131:46259)
0620 20:14:33.825306 (+ 11809us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:33.825314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:33.825344 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:14:33.825433 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:33.825441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:33.825680 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:33.825902 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:33.826306 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.827842 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.830070 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.831615 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.831762 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.832018 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:33.832209 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 20:14:33.832385 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11634,"thread_start_us":11443,"threads_started":1}
I20260620 20:14:33.833578 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.813475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.4.169.129:44541)
0620 20:14:33.822817 (+ 9342us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:33.822829 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:33.824264 (+ 1435us) server_negotiation.cc:407] Connection header received
0620 20:14:33.824500 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:33.824509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:33.824817 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:33.825648 (+ 831us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:33.825894 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.827261 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.830309 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.831764 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.831888 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.832093 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:33.833223 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0620 20:14:33.833347 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9086,"thread_start_us":9978,"threads_started":1}
I20260620 20:14:33.833123 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:33.811912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:44541 (local address 127.0.0.1:44792)
0620 20:14:33.824153 (+ 12241us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:33.824173 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:33.824332 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:33.825076 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:33.825081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:33.825108 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:33.825473 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.825485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.827595 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.827601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.830134 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:33.830147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:33.832262 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:33.832268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:33.832447 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:33.832482 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:14:33.832592 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:14:33.832734 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12097,"mutex_wait_us":24,"thread_start_us":13302,"threads_started":1}
I20260620 20:14:33.837792 24223 tablet_service.cc:1467] Processing CreateTablet for tablet 05e71a3f0cd74e6997e7d65a0002cdba (DEFAULT_TABLE table=test-table [id=5b817ff0fd26422ea585ab92547e27e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:33.837792 24491 tablet_service.cc:1467] Processing CreateTablet for tablet 05e71a3f0cd74e6997e7d65a0002cdba (DEFAULT_TABLE table=test-table [id=5b817ff0fd26422ea585ab92547e27e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:33.839377 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e71a3f0cd74e6997e7d65a0002cdba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:33.839376 24223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e71a3f0cd74e6997e7d65a0002cdba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:33.843946 24575 tablet_bootstrap.cc:492] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: Bootstrap starting.
I20260620 20:14:33.847231 24575 tablet_bootstrap.cc:654] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:33.848284 24575 log.cc:824] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:33.851367 24575 tablet_bootstrap.cc:492] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: No bootstrap required, opened a new log
I20260620 20:14:33.851640 24575 ts_tablet_manager.cc:1397] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 20:14:33.854823 24578 tablet_bootstrap.cc:492] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: Bootstrap starting.
I20260620 20:14:33.855628 24577 tablet_bootstrap.cc:492] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: Bootstrap starting.
I20260620 20:14:33.859570 24577 tablet_bootstrap.cc:654] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:33.860931 24577 log.cc:824] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:33.861706 24578 tablet_bootstrap.cc:654] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:33.862346 24575 raft_consensus.cc:348] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } }
I20260620 20:14:33.862910 24575 raft_consensus.cc:374] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:33.863070 24575 raft_consensus.cc:729] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41975959a1374a72bc088c8a3eb75ea2, State: Initialized, Role: FOLLOWER
I20260620 20:14:33.863147 24578 log.cc:824] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:33.863754 24575 consensus_queue.cc:260] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } }
I20260620 20:14:33.864668 24577 tablet_bootstrap.cc:492] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: No bootstrap required, opened a new log
I20260620 20:14:33.865033 24577 ts_tablet_manager.cc:1397] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 20:14:33.866329 24578 tablet_bootstrap.cc:492] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: No bootstrap required, opened a new log
I20260620 20:14:33.866643 24578 ts_tablet_manager.cc:1397] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 20:14:33.871234 24422 heartbeater.cc:503] Master 127.4.169.190:38417 was elected leader, sending a full tablet report...
I20260620 20:14:33.872089 24575 ts_tablet_manager.cc:1428] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260620 20:14:33.875882 24578 raft_consensus.cc:348] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } }
I20260620 20:14:33.876382 24578 raft_consensus.cc:374] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:33.876523 24578 raft_consensus.cc:729] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bc55f1e4a543878ac41816e44ea10c, State: Initialized, Role: FOLLOWER
I20260620 20:14:33.877251 24578 consensus_queue.cc:260] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } }
I20260620 20:14:33.877821 24577 raft_consensus.cc:348] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } }
I20260620 20:14:33.878381 24577 raft_consensus.cc:374] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:33.878551 24556 heartbeater.cc:503] Master 127.4.169.190:38417 was elected leader, sending a full tablet report...
I20260620 20:14:33.878618 24577 raft_consensus.cc:729] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1c0aafbb254637802b002b22c4a404, State: Initialized, Role: FOLLOWER
I20260620 20:14:33.879381 24578 ts_tablet_manager.cc:1428] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260620 20:14:33.879283 24577 consensus_queue.cc:260] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } }
I20260620 20:14:33.881124 24577 ts_tablet_manager.cc:1428] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260620 20:14:33.908488 24289 tablet.cc:2369] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:33.945856 24423 tablet.cc:2369] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:33.989382 24557 tablet.cc:2369] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:34.044509 24581 raft_consensus.cc:482] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:34.044832 24581 raft_consensus.cc:504] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } }
I20260620 20:14:34.046847 24581 leader_election.cc:288] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6bc55f1e4a543878ac41816e44ea10c (127.4.169.131:46259), 3a1c0aafbb254637802b002b22c4a404 (127.4.169.129:44541)
I20260620 20:14:34.054014 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:34.047381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:56313 (local address 127.4.169.131:46259)
0620 20:14:34.047607 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:34.047612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:34.048363 (+ 751us) server_negotiation.cc:407] Connection header received
0620 20:14:34.048668 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:34.048673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:34.048735 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:34.048831 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:34.049482 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.050235 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.052099 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.052938 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.053132 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:34.053210 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:34.053603 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 20:14:34.053765 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:14:34.054085 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:34.047383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:53353 (local address 127.4.169.129:44541)
0620 20:14:34.047601 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:34.047608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:34.048616 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 20:14:34.048891 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:34.048895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:34.048957 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:34.049064 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:34.049705 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.050504 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.052237 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.053120 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.053314 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:34.053365 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:34.053744 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 20:14:34.053872 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 20:14:34.054220 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:34.047324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:46259 (local address 127.4.169.130:56313)
0620 20:14:34.048224 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:34.048253 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:34.048492 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:34.048857 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:34.048862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:34.048895 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:34.049329 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:34.049339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.050364 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.050372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:34.051918 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:34.051931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.053162 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.053168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:34.053356 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:34.053417 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 20:14:34.053527 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:14:34.053744 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":40,"thread_start_us":485,"threads_started":1}
I20260620 20:14:34.054265 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:34.047818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:44541 (local address 127.4.169.130:53353)
0620 20:14:34.048499 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:34.048517 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:34.048734 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:34.049080 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:34.049084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:34.049112 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:34.049547 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:34.049556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.050638 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.050643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:34.052076 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:34.052094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.053365 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.053370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:34.053577 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:34.053622 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 20:14:34.053704 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:14:34.053799 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":24,"thread_start_us":317,"threads_started":1}
I20260620 20:14:34.055341 24511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e71a3f0cd74e6997e7d65a0002cdba" candidate_uuid: "41975959a1374a72bc088c8a3eb75ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bc55f1e4a543878ac41816e44ea10c" is_pre_election: true
I20260620 20:14:34.055315 24243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e71a3f0cd74e6997e7d65a0002cdba" candidate_uuid: "41975959a1374a72bc088c8a3eb75ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1c0aafbb254637802b002b22c4a404" is_pre_election: true
I20260620 20:14:34.055828 24243 raft_consensus.cc:2454] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41975959a1374a72bc088c8a3eb75ea2 in term 0.
I20260620 20:14:34.055827 24511 raft_consensus.cc:2454] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41975959a1374a72bc088c8a3eb75ea2 in term 0.
I20260620 20:14:34.056475 24313 leader_election.cc:302] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41975959a1374a72bc088c8a3eb75ea2, c6bc55f1e4a543878ac41816e44ea10c; no voters:
I20260620 20:14:34.056896 24581 raft_consensus.cc:2781] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:34.057039 24581 raft_consensus.cc:482] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:34.057153 24581 raft_consensus.cc:3037] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:34.059526 24581 raft_consensus.cc:504] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } }
I20260620 20:14:34.060243 24581 leader_election.cc:288] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [CANDIDATE]: Term 1 election: Requested vote from peers c6bc55f1e4a543878ac41816e44ea10c (127.4.169.131:46259), 3a1c0aafbb254637802b002b22c4a404 (127.4.169.129:44541)
I20260620 20:14:34.060742 24511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e71a3f0cd74e6997e7d65a0002cdba" candidate_uuid: "41975959a1374a72bc088c8a3eb75ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bc55f1e4a543878ac41816e44ea10c"
I20260620 20:14:34.060770 24243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e71a3f0cd74e6997e7d65a0002cdba" candidate_uuid: "41975959a1374a72bc088c8a3eb75ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1c0aafbb254637802b002b22c4a404"
I20260620 20:14:34.060974 24243 raft_consensus.cc:3037] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:34.060966 24511 raft_consensus.cc:3037] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:34.063301 24243 raft_consensus.cc:2454] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41975959a1374a72bc088c8a3eb75ea2 in term 1.
I20260620 20:14:34.063308 24511 raft_consensus.cc:2454] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41975959a1374a72bc088c8a3eb75ea2 in term 1.
I20260620 20:14:34.063833 24312 leader_election.cc:302] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a1c0aafbb254637802b002b22c4a404, 41975959a1374a72bc088c8a3eb75ea2; no voters:
I20260620 20:14:34.064199 24581 raft_consensus.cc:2781] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:34.064738 24581 raft_consensus.cc:686] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 1 LEADER]: Becoming Leader. State: Replica: 41975959a1374a72bc088c8a3eb75ea2, State: Running, Role: LEADER
I20260620 20:14:34.065227 24581 consensus_queue.cc:237] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } }
I20260620 20:14:34.069936 24098 catalog_manager.cc:5495] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41975959a1374a72bc088c8a3eb75ea2 (127.4.169.130). New cstate: current_term: 1 leader_uuid: "41975959a1374a72bc088c8a3eb75ea2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41975959a1374a72bc088c8a3eb75ea2" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 40031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 } health_report { overall_health: UNKNOWN } } }
I20260620 20:14:34.119117 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:34.113341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.4.169.130:40031)
0620 20:14:34.113550 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:34.113557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:34.113660 (+ 103us) server_negotiation.cc:407] Connection header received
0620 20:14:34.113919 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:34.113923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:34.114013 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:34.114175 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:34.114682 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.115409 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.116814 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.117241 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.117385 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:34.117924 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:34.118481 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:34.118674 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:34.118726 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 20:14:34.118814 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:14:34.128252 24243 raft_consensus.cc:1261] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [term 1 FOLLOWER]: Refusing update from remote peer 41975959a1374a72bc088c8a3eb75ea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:34.128252 24511 raft_consensus.cc:1261] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [term 1 FOLLOWER]: Refusing update from remote peer 41975959a1374a72bc088c8a3eb75ea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:34.129521 24586 consensus_queue.cc:1035] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a1c0aafbb254637802b002b22c4a404" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:34.129899 24581 consensus_queue.cc:1035] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bc55f1e4a543878ac41816e44ea10c" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:34.159741 24099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:34.160058 24099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:34.164250 24099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bffae9b1ee7748788b9ae95f70e7a110: Sending DeleteTablet for 3 replicas of tablet 05e71a3f0cd74e6997e7d65a0002cdba
I20260620 20:14:34.165719 24357 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e71a3f0cd74e6997e7d65a0002cdba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:34 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:55366
I20260620 20:14:34.165930 24491 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e71a3f0cd74e6997e7d65a0002cdba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:34 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:57236
I20260620 20:14:34.166560 24223 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e71a3f0cd74e6997e7d65a0002cdba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:34 UTC) from {username='kudu', principal='kudu/127.4.169.190@KRBTEST.COM'} at 127.0.0.1:44792
I20260620 20:14:34.166988 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 24160
I20260620 20:14:34.167850 24599 tablet_replica.cc:330] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: stopping tablet replica
I20260620 20:14:34.168449 24599 raft_consensus.cc:2229] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:34.169482 24599 raft_consensus.cc:2258] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:34.172474 24599 ts_tablet_manager.cc:1905] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:34.176857 24601 tablet_replica.cc:330] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: stopping tablet replica
I20260620 20:14:34.178367 24601 raft_consensus.cc:2229] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:34.181234 24601 raft_consensus.cc:2258] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:34.181254 24599 ts_tablet_manager.cc:1918] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:34.181417 24599 log.cc:1196] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/05e71a3f0cd74e6997e7d65a0002cdba
I20260620 20:14:34.182072 24599 ts_tablet_manager.cc:1939] T 05e71a3f0cd74e6997e7d65a0002cdba P 41975959a1374a72bc088c8a3eb75ea2: Deleting consensus metadata
I20260620 20:14:34.184141 24086 catalog_manager.cc:4841] TS 41975959a1374a72bc088c8a3eb75ea2 (127.4.169.130:40031): tablet 05e71a3f0cd74e6997e7d65a0002cdba (table test-table [id=5b817ff0fd26422ea585ab92547e27e6]) successfully deleted
I20260620 20:14:34.185519 24601 ts_tablet_manager.cc:1905] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:34.190898 24600 tablet_replica.cc:330] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: stopping tablet replica
I20260620 20:14:34.191376 24600 raft_consensus.cc:2229] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [term 1 FOLLOWER]: Raft consensus shutting down.
Jun 20 20:14:34 dist-test-slave-4bwq krb5kdc[24054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.129@KRBTEST.COM
I20260620 20:14:34.191962 24600 raft_consensus.cc:2258] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:34.195888 24601 ts_tablet_manager.cc:1918] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:34.196314 24601 log.cc:1196] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/05e71a3f0cd74e6997e7d65a0002cdba
I20260620 20:14:34.197065 24601 ts_tablet_manager.cc:1939] T 05e71a3f0cd74e6997e7d65a0002cdba P 3a1c0aafbb254637802b002b22c4a404: Deleting consensus metadata
I20260620 20:14:34.199268 24085 catalog_manager.cc:4841] TS 3a1c0aafbb254637802b002b22c4a404 (127.4.169.129:44541): tablet 05e71a3f0cd74e6997e7d65a0002cdba (table test-table [id=5b817ff0fd26422ea585ab92547e27e6]) successfully deleted
I20260620 20:14:34.201241 24600 ts_tablet_manager.cc:1905] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:34.202229 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:34.169209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.4.169.129:44541)
0620 20:14:34.169455 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:34.169464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:34.177534 (+ 8070us) server_negotiation.cc:407] Connection header received
0620 20:14:34.178685 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:34.178691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:34.178787 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:34.178864 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:34.183046 (+ 4182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.184158 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.186116 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.186831 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.186963 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:34.193219 (+ 6256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:34.193236 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:34.193257 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:34.193304 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:34.197779 (+ 4475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:34.198955 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:34.199114 (+ 159us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:34.199120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:34.199310 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:34.199826 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:34.199832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:34.199836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:34.200452 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:34.200783 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:34.201554 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 20:14:34.201807 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:14:34.204268 24283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:34.209846 24600 ts_tablet_manager.cc:1918] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:34.210035 24600 log.cc:1196] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/05e71a3f0cd74e6997e7d65a0002cdba
I20260620 20:14:34.210786 24600 ts_tablet_manager.cc:1939] T 05e71a3f0cd74e6997e7d65a0002cdba P c6bc55f1e4a543878ac41816e44ea10c: Deleting consensus metadata
I20260620 20:14:34.212846 24086 catalog_manager.cc:4841] TS c6bc55f1e4a543878ac41816e44ea10c (127.4.169.131:46259): tablet 05e71a3f0cd74e6997e7d65a0002cdba (table test-table [id=5b817ff0fd26422ea585ab92547e27e6]) successfully deleted
I20260620 20:14:34.301319 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 24160
I20260620 20:14:34.320533 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 24294
Jun 20 20:14:34 dist-test-slave-4bwq krb5kdc[24054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.130@KRBTEST.COM
I20260620 20:14:34.338548 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:34.321568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.4.169.130:40031)
0620 20:14:34.321792 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:34.321799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:34.321820 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:14:34.322024 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:34.322029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:34.322098 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:34.322224 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:34.323756 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.324663 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.326075 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.326721 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.326899 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:34.331827 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:34.331843 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:34.331858 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:34.331917 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:34.335504 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:34.336418 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:34.336429 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:34.336432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:34.336612 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:34.337035 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:34.337038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:34.337040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:34.337412 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:34.337657 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:34.338015 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 20:14:34.338188 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:14:34.339294 24417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:34.457131 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 24294
I20260620 20:14:34.476218 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 24428
Jun 20 20:14:34 dist-test-slave-4bwq krb5kdc[24054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.131@KRBTEST.COM
I20260620 20:14:34.493495 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:34.477084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.4.169.131:46259)
0620 20:14:34.477300 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:34.477308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:34.477332 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:14:34.477561 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:34.477567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:34.477659 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:34.477770 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:34.479377 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.480184 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.481491 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:34.481918 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:34.482060 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:34.487290 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:34.487308 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:34.487325 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:34.487367 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:34.490587 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:34.491416 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:34.491424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:34.491427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:34.491608 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:34.491952 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:34.491955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:34.491957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:34.492330 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:34.492589 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:34.492891 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 20:14:34.493176 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:14:34.494216 24551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:34.587273 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 24428
I20260620 20:14:34.604655 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 24067
I20260620 20:14:34.605818 24129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:34.724164 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 24067
2026-06-20T20:14:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4556 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T20:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:34Z Disabled control of system clock
I20260620 20:14:34.805461 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41545
--webserver_interface=127.4.169.190
--webserver_port=0
--builtin_ntp_servers=127.4.169.148:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:41545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:14:35.167379 24610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:35.167766 24610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:35.167927 24610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:35.175535 24610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:35.175719 24610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:35.175824 24610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:35.175943 24610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:35.176024 24610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:35.185724 24610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:41545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:35.187186 24610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:35.189075 24610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:35.201792 24615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:35.202131 24616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:35.202353 24618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:35.202909 24610 server_base.cc:1034] running on GCE node
I20260620 20:14:35.203750 24610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:35.205370 24610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:35.206602 24610 hybrid_clock.cc:648] HybridClock initialized: now 1781986475206554 us; error 75 us; skew 500 ppm
I20260620 20:14:35.206993 24610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:35.209364 24610 webserver.cc:460] Webserver started at http://127.4.169.190:33237/ using document root <none> and password file <none>
I20260620 20:14:35.210104 24610 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:35.210255 24610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:35.210647 24610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:35.213872 24610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "8f37d4f0cf584640b59d24e8859bd21b"
format_stamp: "Formatted at 2026-06-20 20:14:35 on dist-test-slave-4bwq"
server_key: "480f8034b87789591061b4be1025f41c"
server_key_iv: "0614135b8a53bd9601d71c263f5e62c3"
server_key_version: "encryptionkey@0"
I20260620 20:14:35.214820 24610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "8f37d4f0cf584640b59d24e8859bd21b"
format_stamp: "Formatted at 2026-06-20 20:14:35 on dist-test-slave-4bwq"
server_key: "480f8034b87789591061b4be1025f41c"
server_key_iv: "0614135b8a53bd9601d71c263f5e62c3"
server_key_version: "encryptionkey@0"
I20260620 20:14:35.220312 24610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:14:35.224155 24624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:35.225593 24610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:14:35.225811 24610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "8f37d4f0cf584640b59d24e8859bd21b"
format_stamp: "Formatted at 2026-06-20 20:14:35 on dist-test-slave-4bwq"
server_key: "480f8034b87789591061b4be1025f41c"
server_key_iv: "0614135b8a53bd9601d71c263f5e62c3"
server_key_version: "encryptionkey@0"
I20260620 20:14:35.226050 24610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:35.242072 24610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:35.242949 24610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:35.243247 24610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:35.261916 24675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:41545 every 8 connection(s)
I20260620 20:14:35.261950 24610 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:41545
I20260620 20:14:35.263433 24610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:35.268249 24676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:35.272478 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 24610
I20260620 20:14:35.272661 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:35.273228 4774 external_mini_cluster.cc:1428] Setting key 6225aa1e925da3733a4b9e943a0fde36
I20260620 20:14:35.281625 24676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b: Bootstrap starting.
I20260620 20:14:35.285385 24676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:35.287011 24676 log.cc:824] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:35.290462 24676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b: No bootstrap required, opened a new log
I20260620 20:14:35.295032 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:35.275379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.4.169.190:41545)
0620 20:14:35.277149 (+ 1770us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:35.277182 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:35.282513 (+ 5331us) server_negotiation.cc:407] Connection header received
0620 20:14:35.284485 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:35.284510 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:35.285019 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:35.285419 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:35.286098 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:35.287914 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:35.289464 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:35.290596 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:35.290786 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:35.291939 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:35.291951 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:35.291971 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:35.292747 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:35.292858 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:35.292886 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:35.293014 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:35.293234 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 20:14:35.293622 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":800,"threads_started":1}
I20260620 20:14:35.300297 24676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37d4f0cf584640b59d24e8859bd21b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41545 } }
I20260620 20:14:35.300652 24676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:35.300727 24676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f37d4f0cf584640b59d24e8859bd21b, State: Initialized, Role: FOLLOWER
I20260620 20:14:35.301221 24676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37d4f0cf584640b59d24e8859bd21b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41545 } }
I20260620 20:14:35.301422 24676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:35.301559 24676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:35.301745 24676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:35.304282 24676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37d4f0cf584640b59d24e8859bd21b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41545 } }
I20260620 20:14:35.304829 24676 leader_election.cc:302] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f37d4f0cf584640b59d24e8859bd21b; no voters:
I20260620 20:14:35.305416 24676 leader_election.cc:288] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:35.305600 24681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:35.307001 24681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [term 1 LEADER]: Becoming Leader. State: Replica: 8f37d4f0cf584640b59d24e8859bd21b, State: Running, Role: LEADER
I20260620 20:14:35.307685 24676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:35.307544 24681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37d4f0cf584640b59d24e8859bd21b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41545 } }
I20260620 20:14:35.310715 24683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f37d4f0cf584640b59d24e8859bd21b. Latest consensus state: current_term: 1 leader_uuid: "8f37d4f0cf584640b59d24e8859bd21b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37d4f0cf584640b59d24e8859bd21b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41545 } } }
I20260620 20:14:35.311179 24683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:35.310693 24682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f37d4f0cf584640b59d24e8859bd21b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37d4f0cf584640b59d24e8859bd21b" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41545 } } }
I20260620 20:14:35.311630 24682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:35.313362 24690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:35.319079 24690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:35.328076 24690 catalog_manager.cc:1347] Generated new cluster ID: 88b8df5ab83a4607a2ea5ac021b873c2
I20260620 20:14:35.328233 24690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:35.354126 24690 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:35.355449 24690 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:35.366636 24690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b: Generated new TSK 0
I20260620 20:14:35.367589 24690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:35.378720 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.129:0
--local_ip_for_outbound_sockets=127.4.169.129
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41545
--builtin_ntp_servers=127.4.169.148:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:14:35.751627 24700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:35.751981 24700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:35.752137 24700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:35.759599 24700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:35.759749 24700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:35.759929 24700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.129
I20260620 20:14:35.769608 24700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.169.129
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.129
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:35.771100 24700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:35.773090 24700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:35.787819 24705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:35.787726 24708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:35.787735 24706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:35.788756 24700 server_base.cc:1034] running on GCE node
I20260620 20:14:35.789543 24700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:35.790680 24700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:35.791910 24700 hybrid_clock.cc:648] HybridClock initialized: now 1781986475791835 us; error 71 us; skew 500 ppm
I20260620 20:14:35.792277 24700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:35.798058 24700 webserver.cc:460] Webserver started at http://127.4.169.129:43275/ using document root <none> and password file <none>
I20260620 20:14:35.798902 24700 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:35.799010 24700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:35.799306 24700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:35.802376 24700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/data/instance:
uuid: "d566860cc4624b4eb19dc0f164e96f71"
format_stamp: "Formatted at 2026-06-20 20:14:35 on dist-test-slave-4bwq"
server_key: "9178764cb8597f851dd7e45598b4c4c8"
server_key_iv: "f9df55582ae356c48299acdb5e2d4f00"
server_key_version: "encryptionkey@0"
I20260620 20:14:35.803325 24700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance:
uuid: "d566860cc4624b4eb19dc0f164e96f71"
format_stamp: "Formatted at 2026-06-20 20:14:35 on dist-test-slave-4bwq"
server_key: "9178764cb8597f851dd7e45598b4c4c8"
server_key_iv: "f9df55582ae356c48299acdb5e2d4f00"
server_key_version: "encryptionkey@0"
I20260620 20:14:35.809218 24700 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 20:14:35.813479 24714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:35.815085 24700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:35.815244 24700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/wal
uuid: "d566860cc4624b4eb19dc0f164e96f71"
format_stamp: "Formatted at 2026-06-20 20:14:35 on dist-test-slave-4bwq"
server_key: "9178764cb8597f851dd7e45598b4c4c8"
server_key_iv: "f9df55582ae356c48299acdb5e2d4f00"
server_key_version: "encryptionkey@0"
I20260620 20:14:35.815452 24700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:35.828179 24700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:35.829031 24700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:35.829311 24700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:35.830466 24700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:35.832048 24700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:35.832144 24700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:35.832229 24700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:35.832285 24700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:35.876302 24700 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.129:41513
I20260620 20:14:35.876392 24826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.129:41513 every 8 connection(s)
I20260620 20:14:35.877744 24700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:35.880344 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 24700
I20260620 20:14:35.880533 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:35.881150 4774 external_mini_cluster.cc:1428] Setting key bb525c66927355af37fdce7fb29eeee2
I20260620 20:14:35.884282 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.130:0
--local_ip_for_outbound_sockets=127.4.169.130
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41545
--builtin_ntp_servers=127.4.169.148:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:14:35.898684 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:35.881847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41545 (local address 127.4.169.129:51847)
0620 20:14:35.883181 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:35.883240 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:35.886165 (+ 2925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:35.887420 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:35.887434 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:35.887595 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:35.888452 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:35.888483 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:35.890268 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:35.890277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:35.892575 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:35.892597 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:35.894018 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:35.894027 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:35.894450 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:35.895419 (+ 969us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:35.895425 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:35.895501 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:35.895631 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:35.896253 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:35.896269 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:35.896277 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:14:35.896489 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 20:14:35.897016 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":570,"threads_started":1}
I20260620 20:14:35.899086 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:35.881087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.129:51847 (local address 127.4.169.190:41545)
0620 20:14:35.881623 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:35.881630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:35.883364 (+ 1734us) server_negotiation.cc:407] Connection header received
0620 20:14:35.886694 (+ 3330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:35.886708 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:35.886837 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:35.886972 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:35.888674 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:35.890049 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:35.892748 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:35.893498 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:35.893684 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:35.895924 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:35.895934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:35.895940 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:35.895972 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:35.896033 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:35.896054 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:35.896191 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:35.898534 (+ 2343us) server_negotiation.cc:299] Negotiation successful
0620 20:14:35.898659 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":263,"threads_started":1}
I20260620 20:14:35.900508 24827 heartbeater.cc:347] Connected to a master server at 127.4.169.190:41545
I20260620 20:14:35.900846 24827 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:35.901641 24827 heartbeater.cc:511] Master 127.4.169.190:41545 requested a full tablet report, sending...
I20260620 20:14:35.903915 24641 ts_manager.cc:194] Registered new tserver with Master: d566860cc4624b4eb19dc0f164e96f71 (127.4.169.129:41513)
I20260620 20:14:35.905467 24641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.129:51847
W20260620 20:14:36.244601 24831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:36.244946 24831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:36.245116 24831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:36.252576 24831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:36.252728 24831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:36.252907 24831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.130
I20260620 20:14:36.262461 24831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.169.130
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.130
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:36.263877 24831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:36.265724 24831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:36.279559 24839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:36.279657 24836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:36.279621 24837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:36.280642 24831 server_base.cc:1034] running on GCE node
I20260620 20:14:36.281283 24831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:36.282259 24831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:36.283495 24831 hybrid_clock.cc:648] HybridClock initialized: now 1781986476283440 us; error 47 us; skew 500 ppm
I20260620 20:14:36.283886 24831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:36.286190 24831 webserver.cc:460] Webserver started at http://127.4.169.130:37401/ using document root <none> and password file <none>
I20260620 20:14:36.286965 24831 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:36.287071 24831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:36.287354 24831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:36.290320 24831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/data/instance:
uuid: "66ff07d607b2457a8ed7bc1f2d61c857"
format_stamp: "Formatted at 2026-06-20 20:14:36 on dist-test-slave-4bwq"
server_key: "4a53f3b433b30715183cc960e57d3aa6"
server_key_iv: "00bbf00f2807186d02e5aa73a326a8cf"
server_key_version: "encryptionkey@0"
I20260620 20:14:36.291234 24831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance:
uuid: "66ff07d607b2457a8ed7bc1f2d61c857"
format_stamp: "Formatted at 2026-06-20 20:14:36 on dist-test-slave-4bwq"
server_key: "4a53f3b433b30715183cc960e57d3aa6"
server_key_iv: "00bbf00f2807186d02e5aa73a326a8cf"
server_key_version: "encryptionkey@0"
I20260620 20:14:36.296576 24831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:14:36.300624 24845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:36.302238 24831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:36.302489 24831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/wal
uuid: "66ff07d607b2457a8ed7bc1f2d61c857"
format_stamp: "Formatted at 2026-06-20 20:14:36 on dist-test-slave-4bwq"
server_key: "4a53f3b433b30715183cc960e57d3aa6"
server_key_iv: "00bbf00f2807186d02e5aa73a326a8cf"
server_key_version: "encryptionkey@0"
I20260620 20:14:36.302817 24831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:36.325860 24831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:36.326855 24831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:36.327225 24831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:36.328334 24831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:36.330076 24831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:36.330180 24831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:36.330344 24831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:36.330463 24831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 20:14:36.367391 24831 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.130:39871
I20260620 20:14:36.367445 24957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.130:39871 every 8 connection(s)
I20260620 20:14:36.368892 24831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:36.375768 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 24831
I20260620 20:14:36.375990 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:36.376515 4774 external_mini_cluster.cc:1428] Setting key 6079d99e19992d3f3216e34acf57108c
I20260620 20:14:36.379839 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.169.131:0
--local_ip_for_outbound_sockets=127.4.169.131
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41545
--builtin_ntp_servers=127.4.169.148:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:14:36.384089 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.371809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:52733 (local address 127.4.169.190:41545)
0620 20:14:36.372072 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:36.372079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:36.373728 (+ 1649us) server_negotiation.cc:407] Connection header received
0620 20:14:36.375753 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:36.375762 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:36.375872 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:36.376020 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:36.377480 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.378490 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.380367 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.381242 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.381411 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.383036 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:36.383046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:36.383055 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:36.383085 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:36.383134 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:36.383154 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:36.383258 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:36.383495 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 20:14:36.383637 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 20:14:36.385061 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.372490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41545 (local address 127.4.169.130:52733)
0620 20:14:36.373521 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:36.373576 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:36.375397 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:36.376316 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:36.376326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:36.376442 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:36.377235 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.377265 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.378680 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.378685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.380194 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.380210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.381496 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.381501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.381745 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.382638 (+ 893us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:36.382641 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:36.382687 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:36.382791 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:36.383266 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:36.383274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:36.383280 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:14:36.383441 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 20:14:36.383797 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":314,"threads_started":1}
I20260620 20:14:36.386924 24958 heartbeater.cc:347] Connected to a master server at 127.4.169.190:41545
I20260620 20:14:36.387341 24958 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:36.388377 24958 heartbeater.cc:511] Master 127.4.169.190:41545 requested a full tablet report, sending...
I20260620 20:14:36.390484 24641 ts_manager.cc:194] Registered new tserver with Master: 66ff07d607b2457a8ed7bc1f2d61c857 (127.4.169.130:39871)
I20260620 20:14:36.391801 24641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.130:52733
W20260620 20:14:36.737306 24961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:36.737607 24961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:36.737696 24961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:36.745407 24961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:36.745540 24961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:36.745656 24961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.169.131
I20260620 20:14:36.755384 24961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.169.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.169.131
--webserver_port=0
--tserver_master_addrs=127.4.169.190:41545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.169.131
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:36.756676 24961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:36.758986 24961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:36.773458 24969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:36.773537 24966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:36.773483 24967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:36.774621 24961 server_base.cc:1034] running on GCE node
I20260620 20:14:36.775374 24961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:36.776432 24961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:36.777719 24961 hybrid_clock.cc:648] HybridClock initialized: now 1781986476777642 us; error 92 us; skew 500 ppm
I20260620 20:14:36.778079 24961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:14:36.780591 24961 webserver.cc:460] Webserver started at http://127.4.169.131:40603/ using document root <none> and password file <none>
I20260620 20:14:36.781391 24961 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:36.781535 24961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:36.781879 24961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:36.785090 24961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/data/instance:
uuid: "a1706689b01340cd94fc2ebdb6a7f4e6"
format_stamp: "Formatted at 2026-06-20 20:14:36 on dist-test-slave-4bwq"
server_key: "0e87d2a7539929dc81a10794ef48d4a2"
server_key_iv: "445989703cf0ca6916b85820b2753ed0"
server_key_version: "encryptionkey@0"
I20260620 20:14:36.786001 24961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance:
uuid: "a1706689b01340cd94fc2ebdb6a7f4e6"
format_stamp: "Formatted at 2026-06-20 20:14:36 on dist-test-slave-4bwq"
server_key: "0e87d2a7539929dc81a10794ef48d4a2"
server_key_iv: "445989703cf0ca6916b85820b2753ed0"
server_key_version: "encryptionkey@0"
I20260620 20:14:36.791719 24961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:14:36.795699 24975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:36.797356 24961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:36.797581 24961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/wal
uuid: "a1706689b01340cd94fc2ebdb6a7f4e6"
format_stamp: "Formatted at 2026-06-20 20:14:36 on dist-test-slave-4bwq"
server_key: "0e87d2a7539929dc81a10794ef48d4a2"
server_key_iv: "445989703cf0ca6916b85820b2753ed0"
server_key_version: "encryptionkey@0"
I20260620 20:14:36.797852 24961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:36.819671 24961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:36.820678 24961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:36.821012 24961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:36.822119 24961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:36.823781 24961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:36.823920 24961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:36.824067 24961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:36.824157 24961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:36.858785 24961 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.131:37541
I20260620 20:14:36.858830 25087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.131:37541 every 8 connection(s)
I20260620 20:14:36.860198 24961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:36.860720 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 24961
I20260620 20:14:36.860909 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:36.861553 4774 external_mini_cluster.cc:1428] Setting key 24adf88d79b303f6ab8b2dbec562fe88
I20260620 20:14:36.874465 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.864737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.131:55307 (local address 127.4.169.190:41545)
0620 20:14:36.864921 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:36.864927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:36.865437 (+ 510us) server_negotiation.cc:407] Connection header received
0620 20:14:36.867469 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:36.867474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:36.867578 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:36.867694 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:36.868897 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.869675 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.871422 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.871834 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.871997 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.873517 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:36.873523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:36.873526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:36.873549 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:36.873581 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:36.873594 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:36.873700 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:36.873931 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 20:14:36.874051 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:14:36.875890 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.864050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:41545 (local address 127.4.169.131:55307)
0620 20:14:36.865281 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:36.865327 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:36.867174 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:36.867944 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:36.867954 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:36.868057 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:36.868718 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.868735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.869859 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.869864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.871262 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.871277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.872097 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.872108 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.872364 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.873151 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:36.873154 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:36.873199 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:36.873293 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:36.873698 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:36.873710 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:36.873721 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:14:36.873920 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 20:14:36.874387 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":387,"threads_started":1}
I20260620 20:14:36.877293 25088 heartbeater.cc:347] Connected to a master server at 127.4.169.190:41545
I20260620 20:14:36.877671 25088 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:36.878584 25088 heartbeater.cc:511] Master 127.4.169.190:41545 requested a full tablet report, sending...
I20260620 20:14:36.880254 24641 ts_manager.cc:194] Registered new tserver with Master: a1706689b01340cd94fc2ebdb6a7f4e6 (127.4.169.131:37541)
I20260620 20:14:36.881104 24641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.169.131:55307
I20260620 20:14:36.890331 4774 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:14:36.900820 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.894796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.4.169.190:41545)
0620 20:14:36.895008 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:36.895013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:36.895442 (+ 429us) server_negotiation.cc:407] Connection header received
0620 20:14:36.895581 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:36.895584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:36.895653 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:36.895748 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:36.896381 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.897134 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.898626 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.899039 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.899175 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.900219 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:36.900224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:36.900227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:36.900247 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:36.900280 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:36.900293 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:36.900365 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:36.900478 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 20:14:36.900575 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:14:36.908838 24827 heartbeater.cc:503] Master 127.4.169.190:41545 was elected leader, sending a full tablet report...
I20260620 20:14:36.905333 24641 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:36.913527 24641 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:36.938036 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.927142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37541 (local address 127.0.0.1:34448)
0620 20:14:36.929312 (+ 2170us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:36.929367 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:36.929615 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:36.930984 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:36.930991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:36.931033 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:36.931412 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.931423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.933121 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.933129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.935190 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.935202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.936547 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.936554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.937171 (+ 617us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.937227 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 20:14:36.937377 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 20:14:36.937613 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1727,"mutex_wait_us":387,"thread_start_us":2855,"threads_started":1}
I20260620 20:14:36.938051 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.927304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.4.169.131:37541)
0620 20:14:36.928465 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:36.928478 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:36.929515 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 20:14:36.930073 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:36.930086 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:36.930431 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:36.930646 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:36.931621 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.932984 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.935394 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.936342 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.936520 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.936847 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:36.937440 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 20:14:36.937654 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":697,"threads_started":1}
I20260620 20:14:36.940645 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.927220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:41513 (local address 127.0.0.1:47246)
0620 20:14:36.930280 (+ 3060us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:36.930301 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:36.930440 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:36.931478 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:36.931483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:36.931507 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:36.932012 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.932023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.934653 (+ 2630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.934662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.937400 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.937412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.939733 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.939743 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.939928 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.939965 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:14:36.940060 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 20:14:36.940240 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2927,"mutex_wait_us":4,"thread_start_us":1459,"threads_started":1}
I20260620 20:14:36.940824 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.927919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.4.169.129:41513)
0620 20:14:36.929334 (+ 1415us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:36.929344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:36.930585 (+ 1241us) server_negotiation.cc:407] Connection header received
0620 20:14:36.930694 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:36.930702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:36.930951 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:36.931189 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:36.932275 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.934444 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.937613 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.939202 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.939330 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.939599 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:36.940253 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 20:14:36.940453 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":1230,"threads_started":1}
I20260620 20:14:36.943019 25023 tablet_service.cc:1467] Processing CreateTablet for tablet fd82439a8dac4c06b6fd56eee689d576 (DEFAULT_TABLE table=test-table [id=7ec83fa01dce4fea8477ed132ad27616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:36.944620 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.931517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.130:39871 (local address 127.0.0.1:36750)
0620 20:14:36.932162 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:36.932183 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:36.932288 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:36.937944 (+ 5656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:36.937949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:36.937979 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:36.938335 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.938347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.940024 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.940032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.942148 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:36.942160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.943907 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.943915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:36.944094 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.944133 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:14:36.944213 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:14:36.944345 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":448,"threads_started":1}
I20260620 20:14:36.944728 25023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd82439a8dac4c06b6fd56eee689d576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:36.945359 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:36.930722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.4.169.130:39871)
0620 20:14:36.937079 (+ 6357us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:36.937087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:36.937113 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:14:36.937192 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:36.937197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:36.937459 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:36.937670 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:36.938581 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.939883 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.942350 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:36.943385 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:36.943529 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:36.943780 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:36.944883 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0620 20:14:36.945100 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6211,"thread_start_us":6078,"threads_started":1}
I20260620 20:14:36.944756 24762 tablet_service.cc:1467] Processing CreateTablet for tablet fd82439a8dac4c06b6fd56eee689d576 (DEFAULT_TABLE table=test-table [id=7ec83fa01dce4fea8477ed132ad27616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:36.946333 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd82439a8dac4c06b6fd56eee689d576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:36.949240 24893 tablet_service.cc:1467] Processing CreateTablet for tablet fd82439a8dac4c06b6fd56eee689d576 (DEFAULT_TABLE table=test-table [id=7ec83fa01dce4fea8477ed132ad27616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:36.950742 24893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd82439a8dac4c06b6fd56eee689d576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:36.960340 25106 tablet_bootstrap.cc:492] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: Bootstrap starting.
I20260620 20:14:36.963037 25108 tablet_bootstrap.cc:492] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: Bootstrap starting.
I20260620 20:14:36.964524 25106 tablet_bootstrap.cc:654] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:36.965553 25107 tablet_bootstrap.cc:492] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: Bootstrap starting.
I20260620 20:14:36.967585 25108 tablet_bootstrap.cc:654] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:36.967826 25106 log.cc:824] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:36.968938 25108 log.cc:824] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:36.969697 25107 tablet_bootstrap.cc:654] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:36.970957 25107 log.cc:824] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:36.971467 25106 tablet_bootstrap.cc:492] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: No bootstrap required, opened a new log
I20260620 20:14:36.971832 25106 ts_tablet_manager.cc:1397] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260620 20:14:36.972203 25108 tablet_bootstrap.cc:492] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: No bootstrap required, opened a new log
I20260620 20:14:36.972501 25108 ts_tablet_manager.cc:1397] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 20:14:36.973734 25107 tablet_bootstrap.cc:492] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: No bootstrap required, opened a new log
I20260620 20:14:36.974011 25107 ts_tablet_manager.cc:1397] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:14:36.982012 25107 raft_consensus.cc:348] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } }
I20260620 20:14:36.982506 25107 raft_consensus.cc:374] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:36.982674 25107 raft_consensus.cc:729] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d566860cc4624b4eb19dc0f164e96f71, State: Initialized, Role: FOLLOWER
I20260620 20:14:36.983261 25106 raft_consensus.cc:348] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } }
I20260620 20:14:36.983271 25108 raft_consensus.cc:348] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } }
I20260620 20:14:36.983467 25107 consensus_queue.cc:260] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } }
I20260620 20:14:36.983767 25106 raft_consensus.cc:374] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:36.983767 25108 raft_consensus.cc:374] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:36.983873 25108 raft_consensus.cc:729] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ff07d607b2457a8ed7bc1f2d61c857, State: Initialized, Role: FOLLOWER
I20260620 20:14:36.983940 25106 raft_consensus.cc:729] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1706689b01340cd94fc2ebdb6a7f4e6, State: Initialized, Role: FOLLOWER
I20260620 20:14:36.984642 25106 consensus_queue.cc:260] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } }
I20260620 20:14:36.984642 25108 consensus_queue.cc:260] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } }
I20260620 20:14:36.985576 25107 ts_tablet_manager.cc:1428] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 20:14:36.986830 24958 heartbeater.cc:503] Master 127.4.169.190:41545 was elected leader, sending a full tablet report...
I20260620 20:14:36.987262 25088 heartbeater.cc:503] Master 127.4.169.190:41545 was elected leader, sending a full tablet report...
I20260620 20:14:36.987558 25108 ts_tablet_manager.cc:1428] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 20:14:36.987803 25106 ts_tablet_manager.cc:1428] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260620 20:14:37.031590 25113 raft_consensus.cc:482] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:37.031975 25113 raft_consensus.cc:504] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } }
I20260620 20:14:37.034373 25113 leader_election.cc:288] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1706689b01340cd94fc2ebdb6a7f4e6 (127.4.169.131:37541), d566860cc4624b4eb19dc0f164e96f71 (127.4.169.129:41513)
I20260620 20:14:37.041826 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:37.035421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:36671 (local address 127.4.169.131:37541)
0620 20:14:37.035621 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:37.035630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:37.036042 (+ 412us) server_negotiation.cc:407] Connection header received
0620 20:14:37.036322 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:37.036329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:37.036408 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:37.036514 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:37.037362 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.038098 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.040105 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.040898 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.041074 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:37.041152 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:37.041436 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 20:14:37.041570 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:14:37.041877 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:37.035139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.169.130:40093 (local address 127.4.169.129:41513)
0620 20:14:37.035401 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:37.035408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:37.036489 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 20:14:37.036624 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:37.036632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:37.036717 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:37.036820 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:37.037413 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.038126 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.040142 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.040905 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.041119 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:37.041200 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:37.041499 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 20:14:37.041629 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:14:37.042016 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:37.034752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.129:41513 (local address 127.4.169.130:40093)
0620 20:14:37.036376 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:37.036391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:37.036501 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:37.036842 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:37.036845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:37.036876 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:37.037223 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:37.037230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.038278 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.038287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:37.039907 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:37.039919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.041163 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.041171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:37.041340 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:37.041362 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:14:37.041461 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:14:37.041610 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1520,"mutex_wait_us":28,"thread_start_us":883,"threads_started":1}
I20260620 20:14:37.042059 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:37.034629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.131:37541 (local address 127.4.169.130:36671)
0620 20:14:37.035925 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:37.035941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:37.036166 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:37.036677 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:37.036682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:37.036708 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:37.037204 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:37.037217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.038224 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.038232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:37.039903 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:37.039915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.041117 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.041124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:37.041277 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:37.041304 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:14:37.041411 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:14:37.041557 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":62,"spinlock_wait_cycles":22784,"thread_start_us":1434,"threads_started":1}
I20260620 20:14:37.043185 24782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd82439a8dac4c06b6fd56eee689d576" candidate_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d566860cc4624b4eb19dc0f164e96f71" is_pre_election: true
I20260620 20:14:37.043180 25043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd82439a8dac4c06b6fd56eee689d576" candidate_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" is_pre_election: true
I20260620 20:14:37.043699 24782 raft_consensus.cc:2454] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ff07d607b2457a8ed7bc1f2d61c857 in term 0.
I20260620 20:14:37.043699 25043 raft_consensus.cc:2454] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ff07d607b2457a8ed7bc1f2d61c857 in term 0.
I20260620 20:14:37.044385 24849 leader_election.cc:302] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 66ff07d607b2457a8ed7bc1f2d61c857, a1706689b01340cd94fc2ebdb6a7f4e6, d566860cc4624b4eb19dc0f164e96f71; no voters:
I20260620 20:14:37.044788 25113 raft_consensus.cc:2781] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:37.044935 25113 raft_consensus.cc:482] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:37.045058 25113 raft_consensus.cc:3037] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:37.047408 25113 raft_consensus.cc:504] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } }
I20260620 20:14:37.047991 25113 leader_election.cc:288] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [CANDIDATE]: Term 1 election: Requested vote from peers a1706689b01340cd94fc2ebdb6a7f4e6 (127.4.169.131:37541), d566860cc4624b4eb19dc0f164e96f71 (127.4.169.129:41513)
I20260620 20:14:37.048489 25043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd82439a8dac4c06b6fd56eee689d576" candidate_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6"
I20260620 20:14:37.048514 24782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd82439a8dac4c06b6fd56eee689d576" candidate_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d566860cc4624b4eb19dc0f164e96f71"
I20260620 20:14:37.048705 25043 raft_consensus.cc:3037] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:37.048722 24782 raft_consensus.cc:3037] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:37.051023 24782 raft_consensus.cc:2454] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ff07d607b2457a8ed7bc1f2d61c857 in term 1.
I20260620 20:14:37.051025 25043 raft_consensus.cc:2454] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ff07d607b2457a8ed7bc1f2d61c857 in term 1.
I20260620 20:14:37.051460 24849 leader_election.cc:302] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ff07d607b2457a8ed7bc1f2d61c857, a1706689b01340cd94fc2ebdb6a7f4e6; no voters:
I20260620 20:14:37.051762 25113 raft_consensus.cc:2781] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:37.052294 25113 raft_consensus.cc:686] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 1 LEADER]: Becoming Leader. State: Replica: 66ff07d607b2457a8ed7bc1f2d61c857, State: Running, Role: LEADER
I20260620 20:14:37.052876 25113 consensus_queue.cc:237] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } }
I20260620 20:14:37.057796 24641 catalog_manager.cc:5495] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66ff07d607b2457a8ed7bc1f2d61c857 (127.4.169.130). New cstate: current_term: 1 leader_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66ff07d607b2457a8ed7bc1f2d61c857" member_type: VOTER last_known_addr { host: "127.4.169.130" port: 39871 } health_report { overall_health: HEALTHY } } }
I20260620 20:14:37.096006 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:37.090360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.4.169.130:39871)
0620 20:14:37.090616 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:37.090622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:37.090641 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:14:37.090738 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:37.090742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:37.090839 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:37.090980 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:37.091379 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.092174 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.093768 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.094230 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.094336 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:37.094833 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:37.095376 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:37.095505 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:37.095589 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 20:14:37.095663 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 20:14:37.104954 24782 raft_consensus.cc:1261] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [term 1 FOLLOWER]: Refusing update from remote peer 66ff07d607b2457a8ed7bc1f2d61c857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:37.104952 25043 raft_consensus.cc:1261] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [term 1 FOLLOWER]: Refusing update from remote peer 66ff07d607b2457a8ed7bc1f2d61c857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:37.106104 25117 consensus_queue.cc:1035] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d566860cc4624b4eb19dc0f164e96f71" member_type: VOTER last_known_addr { host: "127.4.169.129" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:37.106523 25113 consensus_queue.cc:1035] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1706689b01340cd94fc2ebdb6a7f4e6" member_type: VOTER last_known_addr { host: "127.4.169.131" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:14:37.112104 25089 tablet.cc:2369] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:37.122226 24959 tablet.cc:2369] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:37.139360 24640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:37.139660 24640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:37.143610 24640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f37d4f0cf584640b59d24e8859bd21b: Sending DeleteTablet for 3 replicas of tablet fd82439a8dac4c06b6fd56eee689d576
I20260620 20:14:37.144938 24762 tablet_service.cc:1514] Processing DeleteTablet for tablet fd82439a8dac4c06b6fd56eee689d576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:37 UTC) from {username='slave'} at 127.0.0.1:47246
I20260620 20:14:37.144995 25023 tablet_service.cc:1514] Processing DeleteTablet for tablet fd82439a8dac4c06b6fd56eee689d576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:37 UTC) from {username='slave'} at 127.0.0.1:34448
I20260620 20:14:37.145254 24893 tablet_service.cc:1514] Processing DeleteTablet for tablet fd82439a8dac4c06b6fd56eee689d576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:37 UTC) from {username='slave'} at 127.0.0.1:36750
I20260620 20:14:37.147540 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 24700
I20260620 20:14:37.155925 25131 tablet_replica.cc:330] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: stopping tablet replica
I20260620 20:14:37.156133 25129 tablet_replica.cc:330] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: stopping tablet replica
I20260620 20:14:37.156457 25131 raft_consensus.cc:2229] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:37.156488 25129 raft_consensus.cc:2229] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:37.157073 25129 raft_consensus.cc:2258] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:37.157276 25131 raft_consensus.cc:2258] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:37.157992 25130 tablet_replica.cc:330] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: stopping tablet replica
I20260620 20:14:37.158567 25130 raft_consensus.cc:2229] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:37.159207 25130 raft_consensus.cc:2258] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:37.160470 25131 ts_tablet_manager.cc:1905] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:37.161623 25129 ts_tablet_manager.cc:1905] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:37.168728 25131 ts_tablet_manager.cc:1918] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:37.168954 25130 ts_tablet_manager.cc:1905] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:37.169113 25131 log.cc:1196] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-1/wal/wals/fd82439a8dac4c06b6fd56eee689d576
I20260620 20:14:37.169412 25129 ts_tablet_manager.cc:1918] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:37.169600 25129 log.cc:1196] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-2/wal/wals/fd82439a8dac4c06b6fd56eee689d576
I20260620 20:14:37.169984 25131 ts_tablet_manager.cc:1939] T fd82439a8dac4c06b6fd56eee689d576 P 66ff07d607b2457a8ed7bc1f2d61c857: Deleting consensus metadata
I20260620 20:14:37.170141 25129 ts_tablet_manager.cc:1939] T fd82439a8dac4c06b6fd56eee689d576 P a1706689b01340cd94fc2ebdb6a7f4e6: Deleting consensus metadata
I20260620 20:14:37.172137 24628 catalog_manager.cc:4841] TS a1706689b01340cd94fc2ebdb6a7f4e6 (127.4.169.131:37541): tablet fd82439a8dac4c06b6fd56eee689d576 (table test-table [id=7ec83fa01dce4fea8477ed132ad27616]) successfully deleted
I20260620 20:14:37.172500 24628 catalog_manager.cc:4841] TS 66ff07d607b2457a8ed7bc1f2d61c857 (127.4.169.130:39871): tablet fd82439a8dac4c06b6fd56eee689d576 (table test-table [id=7ec83fa01dce4fea8477ed132ad27616]) successfully deleted
I20260620 20:14:37.174060 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:37.149408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.4.169.129:41513)
0620 20:14:37.149655 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:37.149664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:37.160636 (+ 10972us) server_negotiation.cc:407] Connection header received
0620 20:14:37.161202 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:37.161208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:37.161334 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:37.161415 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:37.162144 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.163280 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.165269 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.165890 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.166036 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:37.172192 (+ 6156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:37.172203 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:37.172221 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:37.173019 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:37.173084 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:37.173114 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:37.173249 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:37.173487 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 20:14:37.173624 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 20:14:37.174887 24822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:37.286711 25130 ts_tablet_manager.cc:1918] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:37.286933 25130 log.cc:1196] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: Deleting WAL directory at /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986152907614-4774-0/minicluster-data/ts-0/wal/wals/fd82439a8dac4c06b6fd56eee689d576
I20260620 20:14:37.287518 25130 ts_tablet_manager.cc:1939] T fd82439a8dac4c06b6fd56eee689d576 P d566860cc4624b4eb19dc0f164e96f71: Deleting consensus metadata
I20260620 20:14:37.289464 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 24700
I20260620 20:14:37.289391 24627 catalog_manager.cc:4841] TS d566860cc4624b4eb19dc0f164e96f71 (127.4.169.129:41513): tablet fd82439a8dac4c06b6fd56eee689d576 (table test-table [id=7ec83fa01dce4fea8477ed132ad27616]) successfully deleted
I20260620 20:14:37.313791 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 24831
I20260620 20:14:37.322225 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:37.314879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.4.169.130:39871)
0620 20:14:37.315071 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:37.315076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:37.315095 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:14:37.315160 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:37.315164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:37.315280 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:37.315366 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:37.315886 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.316777 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.318713 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.319182 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.319290 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:37.320599 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:37.320610 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:37.320626 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:37.321300 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:37.321365 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:37.321392 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:37.321481 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:37.321671 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 20:14:37.321770 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:14:37.323024 24953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:37.435134 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 24831
I20260620 20:14:37.454254 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 24961
I20260620 20:14:37.461289 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:37.455094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.4.169.131:37541)
0620 20:14:37.455279 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:37.455284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:37.455316 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:14:37.455454 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:37.455459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:37.455544 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:37.455632 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:37.456007 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.456760 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.458029 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:37.458671 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:37.458786 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:37.459895 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:37.459905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:37.459919 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:37.460501 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:37.460553 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:37.460579 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:37.460682 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:37.460819 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:14:37.460920 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:14:37.462003 25083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:37.551713 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 24961
I20260620 20:14:37.568614 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 24610
I20260620 20:14:37.569799 24671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:37.682993 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 24610
2026-06-20T20:14:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2961 ms)
[----------] 6 tests from EncryptionPolicyTest (22207 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:37 dist-test-slave-4bwq krb5kdc[25139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:37 dist-test-slave-4bwq krb5kdc[25139](info): set up 2 sockets
Jun 20 20:14:37 dist-test-slave-4bwq krb5kdc[25139](info): commencing operation
krb5kdc: starting...
W20260620 20:14:38.815333 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:38 dist-test-slave-4bwq krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:39.094767 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:39.450330 25152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:39.450675 25152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:39.450877 25152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:39.458360 25152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:39.458539 25152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:39.458658 25152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:39.458742 25152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:39.458822 25152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:39.469547 25152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:39.471174 25152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:39.472692 25152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:39.485916 25160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:39.486001 25157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:39.485975 25158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:39.487294 25152 server_base.cc:1034] running on GCE node
I20260620 20:14:39.488224 25152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:39.490157 25152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:39.491432 25152 hybrid_clock.cc:648] HybridClock initialized: now 1781986479491372 us; error 59 us; skew 500 ppm
I20260620 20:14:39.491868 25152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:39 dist-test-slave-4bwq krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:39.495468 25152 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:39.497232 25152 webserver.cc:460] Webserver started at http://127.0.0.1:34251/ using document root <none> and password file <none>
I20260620 20:14:39.498013 25152 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:39.498175 25152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:39.498565 25152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:39.501941 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "ebaf7281a4fb4b6fb0e805111e20808d"
format_stamp: "Formatted at 2026-06-20 20:14:39 on dist-test-slave-4bwq"
server_key: "07159d391216013b4dd7c6260011c2dc"
server_key_iv: "9294618a15d08122d69a230c19d8d7a5"
server_key_version: "encryptionkey@0"
I20260620 20:14:39.502856 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "ebaf7281a4fb4b6fb0e805111e20808d"
format_stamp: "Formatted at 2026-06-20 20:14:39 on dist-test-slave-4bwq"
server_key: "07159d391216013b4dd7c6260011c2dc"
server_key_iv: "9294618a15d08122d69a230c19d8d7a5"
server_key_version: "encryptionkey@0"
I20260620 20:14:39.508808 25152 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260620 20:14:39.512876 25167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:39.514662 25152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:39.514879 25152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "ebaf7281a4fb4b6fb0e805111e20808d"
format_stamp: "Formatted at 2026-06-20 20:14:39 on dist-test-slave-4bwq"
server_key: "07159d391216013b4dd7c6260011c2dc"
server_key_iv: "9294618a15d08122d69a230c19d8d7a5"
server_key_version: "encryptionkey@0"
I20260620 20:14:39.515134 25152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:39.537484 25152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:39.541929 25152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:39.542235 25152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:39.561180 25152 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45877
I20260620 20:14:39.561199 25218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45877 every 8 connection(s)
I20260620 20:14:39.562840 25152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:39.567492 25219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:39.571866 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 25152
I20260620 20:14:39.572086 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:39.572767 4774 external_mini_cluster.cc:1428] Setting key 2d3fb713383c2b1167fdec0c2a3be8f6
I20260620 20:14:39.583215 25219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d: Bootstrap starting.
I20260620 20:14:39.587433 25219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:39.588625 25219 log.cc:824] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:14:39 dist-test-slave-4bwq krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:14:39.592705 25219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d: No bootstrap required, opened a new log
I20260620 20:14:39.601058 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:39.575050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45185 (local address 127.0.0.1:45877)
0620 20:14:39.575906 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:39.575922 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:39.575978 (+ 56us) server_negotiation.cc:407] Connection header received
0620 20:14:39.578107 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:39.578145 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:39.578852 (+ 707us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:39.579390 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:39.580785 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:39.583010 (+ 2225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:39.584509 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:39.585800 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:39.586001 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:39.590776 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:39.590788 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:39.590807 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:39.590854 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:39.595612 (+ 4758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:39.596619 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:39.596629 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:39.596639 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:39.596894 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:39.597366 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:39.597371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:39.597374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:39.598183 (+ 809us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:39.598571 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:39.599088 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 20:14:39.599495 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":353,"threads_started":1}
I20260620 20:14:39.603627 25219 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebaf7281a4fb4b6fb0e805111e20808d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } }
I20260620 20:14:39.604132 25219 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:39.604241 25219 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebaf7281a4fb4b6fb0e805111e20808d, State: Initialized, Role: FOLLOWER
I20260620 20:14:39.604951 25219 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebaf7281a4fb4b6fb0e805111e20808d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } }
I20260620 20:14:39.605223 25219 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:39.605363 25219 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:39.605563 25219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:39.605778 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:39.608026 25219 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebaf7281a4fb4b6fb0e805111e20808d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } }
I20260620 20:14:39.608580 25219 leader_election.cc:302] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebaf7281a4fb4b6fb0e805111e20808d; no voters:
I20260620 20:14:39.609329 25219 leader_election.cc:288] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:39.609524 25226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:39.610635 25226 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [term 1 LEADER]: Becoming Leader. State: Replica: ebaf7281a4fb4b6fb0e805111e20808d, State: Running, Role: LEADER
I20260620 20:14:39.611160 25226 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebaf7281a4fb4b6fb0e805111e20808d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } }
I20260620 20:14:39.613933 25229 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebaf7281a4fb4b6fb0e805111e20808d. Latest consensus state: current_term: 1 leader_uuid: "ebaf7281a4fb4b6fb0e805111e20808d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebaf7281a4fb4b6fb0e805111e20808d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } } }
I20260620 20:14:39.614629 25229 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:39.614748 25228 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebaf7281a4fb4b6fb0e805111e20808d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebaf7281a4fb4b6fb0e805111e20808d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } } }
I20260620 20:14:39.615172 25228 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:39.614432 25219 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:39.628342 25233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:39.634028 25233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:39.650992 25233 catalog_manager.cc:1347] Generated new cluster ID: f2034b08a8ed426cbe95f09d7b8fe32e
I20260620 20:14:39.651122 25233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:39.652382 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:39.622459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46185 (local address 127.0.0.1:45877)
0620 20:14:39.622715 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:39.622725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:39.627074 (+ 4349us) server_negotiation.cc:407] Connection header received
0620 20:14:39.627838 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:39.627848 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:39.627962 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:39.628069 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:39.631054 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:39.632731 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:39.634907 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:39.635933 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:39.636084 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:39.639945 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:39.639953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:39.639958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:39.640011 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:39.646349 (+ 6338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:39.647851 (+ 1502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:39.647860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:39.647869 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:39.648086 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:39.648895 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:39.648903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:39.648909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:39.649500 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:39.649834 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:39.651546 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0620 20:14:39.651826 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:14:39.664669 25233 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:39.666729 25233 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:39.673970 25233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebaf7281a4fb4b6fb0e805111e20808d: Generated new TSK 0
I20260620 20:14:39.675149 25233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:39.684847 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25152
I20260620 20:14:39.686821 25214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:39.788887 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 25152
W20260620 20:14:39.806996 25137 connection.cc:535] client connection to 127.0.0.1:45877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45877): Connection reset by peer (error 104)
2026-06-20T20:14:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2097 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:39 dist-test-slave-4bwq krb5kdc[25251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:39 dist-test-slave-4bwq krb5kdc[25251](info): set up 2 sockets
Jun 20 20:14:39 dist-test-slave-4bwq krb5kdc[25251](info): commencing operation
krb5kdc: starting...
W20260620 20:14:40.907248 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:41 dist-test-slave-4bwq krb5kdc[25251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:41.175415 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:41.523717 25264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:41.524101 25264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:41.524277 25264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:41.532105 25264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:41.532230 25264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:41.532348 25264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:41.532459 25264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:41.532545 25264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:41.542739 25264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:41.544098 25264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:41.545418 25264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:41.557982 25269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:41.557965 25272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:41.557971 25270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:41.559134 25264 server_base.cc:1034] running on GCE node
I20260620 20:14:41.559986 25264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:41.561558 25264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:41.562803 25264 hybrid_clock.cc:648] HybridClock initialized: now 1781986481562769 us; error 58 us; skew 500 ppm
I20260620 20:14:41.563194 25264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:41 dist-test-slave-4bwq krb5kdc[25251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:41.566926 25264 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:41.568646 25264 webserver.cc:460] Webserver started at http://127.0.0.1:45655/ using document root <none> and password file <none>
I20260620 20:14:41.569422 25264 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:41.569589 25264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:41.569918 25264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:41.573220 25264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "311793fcc29c44978c2a7269bea4f1ac"
format_stamp: "Formatted at 2026-06-20 20:14:41 on dist-test-slave-4bwq"
server_key: "df1d1ad366866d81ac38d620ab4cd077"
server_key_iv: "930b9fceacfa830c1aee6a3894073ae9"
server_key_version: "encryptionkey@0"
I20260620 20:14:41.574035 25264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "311793fcc29c44978c2a7269bea4f1ac"
format_stamp: "Formatted at 2026-06-20 20:14:41 on dist-test-slave-4bwq"
server_key: "df1d1ad366866d81ac38d620ab4cd077"
server_key_iv: "930b9fceacfa830c1aee6a3894073ae9"
server_key_version: "encryptionkey@0"
I20260620 20:14:41.579417 25264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 20:14:41.583022 25279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:41.584479 25264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:41.584682 25264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "311793fcc29c44978c2a7269bea4f1ac"
format_stamp: "Formatted at 2026-06-20 20:14:41 on dist-test-slave-4bwq"
server_key: "df1d1ad366866d81ac38d620ab4cd077"
server_key_iv: "930b9fceacfa830c1aee6a3894073ae9"
server_key_version: "encryptionkey@0"
I20260620 20:14:41.584903 25264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:41.618373 25264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:41.623327 25264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:41.623651 25264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:41.643353 25264 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44629
I20260620 20:14:41.643391 25330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44629 every 8 connection(s)
I20260620 20:14:41.644784 25264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:41.648653 25331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:41.651672 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 25264
I20260620 20:14:41.651851 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:41.652370 4774 external_mini_cluster.cc:1428] Setting key f53730f94cac47ab8612fc0a8166fa5d
I20260620 20:14:41.662058 25331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac: Bootstrap starting.
I20260620 20:14:41.665680 25331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:41.666913 25331 log.cc:824] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:41.667305 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:41.654679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43565 (local address 127.0.0.1:44629)
0620 20:14:41.655484 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:41.655496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:41.655543 (+ 47us) server_negotiation.cc:407] Connection header received
0620 20:14:41.657254 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:41.657279 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:41.657713 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:41.658199 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:41.659431 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:41.660830 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:41.662207 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:41.663054 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:41.663277 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:41.664694 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:41.664703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:41.664720 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:41.665288 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:41.665372 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:41.665398 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:41.665578 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:41.665891 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 20:14:41.666111 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":370,"threads_started":1}
I20260620 20:14:41.670102 25331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac: No bootstrap required, opened a new log
I20260620 20:14:41.671522 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:41.681118 25331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311793fcc29c44978c2a7269bea4f1ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260620 20:14:41.681607 25331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:41.681731 25331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311793fcc29c44978c2a7269bea4f1ac, State: Initialized, Role: FOLLOWER
I20260620 20:14:41.682560 25331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311793fcc29c44978c2a7269bea4f1ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260620 20:14:41.682916 25331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:41.683089 25331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:41.683300 25331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:41.684734 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:41.677417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50669 (local address 127.0.0.1:44629)
0620 20:14:41.677661 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:41.677674 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:41.678088 (+ 414us) server_negotiation.cc:407] Connection header received
0620 20:14:41.678239 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:41.678245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:41.678334 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:41.678487 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:41.679014 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:41.680238 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:41.681791 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:41.682474 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:41.682627 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:41.683878 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:41.683884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:41.683888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:41.683918 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:41.683992 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:41.684016 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:41.684192 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:41.684304 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 20:14:41.684369 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:14:41.686700 25331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311793fcc29c44978c2a7269bea4f1ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260620 20:14:41.687224 25331 leader_election.cc:302] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 311793fcc29c44978c2a7269bea4f1ac; no voters:
I20260620 20:14:41.687767 25331 leader_election.cc:288] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:41.687932 25341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:41.689648 25341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [term 1 LEADER]: Becoming Leader. State: Replica: 311793fcc29c44978c2a7269bea4f1ac, State: Running, Role: LEADER
I20260620 20:14:41.690330 25331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:41.690331 25341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311793fcc29c44978c2a7269bea4f1ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260620 20:14:41.696609 25343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311793fcc29c44978c2a7269bea4f1ac. Latest consensus state: current_term: 1 leader_uuid: "311793fcc29c44978c2a7269bea4f1ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311793fcc29c44978c2a7269bea4f1ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } } }
I20260620 20:14:41.697042 25343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:41.697882 25342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "311793fcc29c44978c2a7269bea4f1ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311793fcc29c44978c2a7269bea4f1ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } } }
I20260620 20:14:41.698252 25342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:41.704028 25351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:41.708642 25351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:41.717792 25351 catalog_manager.cc:1347] Generated new cluster ID: 78757f948c9c469a9a07a5905fe339d8
I20260620 20:14:41.717940 25351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:41.735272 25351 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:41.736629 25351 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:41.748912 25351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 311793fcc29c44978c2a7269bea4f1ac: Generated new TSK 0
I20260620 20:14:41.749825 25351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:41.755779 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25264
W20260620 20:14:41.757038 25326 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43565
W20260620 20:14:41.757882 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25264: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:14:41.758091 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 25264
W20260620 20:14:41.772547 25249 connection.cc:535] client connection to 127.0.0.1:44629 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44629): Connection reset by peer (error 104)
2026-06-20T20:14:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:41 dist-test-slave-4bwq krb5kdc[25362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:41 dist-test-slave-4bwq krb5kdc[25362](info): set up 2 sockets
Jun 20 20:14:41 dist-test-slave-4bwq krb5kdc[25362](info): commencing operation
krb5kdc: starting...
W20260620 20:14:42.855086 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:43 dist-test-slave-4bwq krb5kdc[25362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:43.113976 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:43.443555 25375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:43.443928 25375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:43.444094 25375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:43.451552 25375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:43.451725 25375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:43.451836 25375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:43.451946 25375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:43.452032 25375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:43.462214 25375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:43.463629 25375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:43.464985 25375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:43.477408 25380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:43.477377 25383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:43.477371 25381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:43.478369 25375 server_base.cc:1034] running on GCE node
I20260620 20:14:43.479255 25375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:43.480765 25375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:43.482028 25375 hybrid_clock.cc:648] HybridClock initialized: now 1781986483481972 us; error 38 us; skew 500 ppm
I20260620 20:14:43.482481 25375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:43 dist-test-slave-4bwq krb5kdc[25362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:43.493332 25375 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:43.494992 25375 webserver.cc:460] Webserver started at http://127.0.0.1:34159/ using document root <none> and password file <none>
I20260620 20:14:43.495918 25375 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:43.496081 25375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:43.496393 25375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:43.499532 25375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "e899d4037b454322a9f4dc645c7abbd4"
format_stamp: "Formatted at 2026-06-20 20:14:43 on dist-test-slave-4bwq"
server_key: "6a8cc594f4ea55ca6ca017eec1f3e8d7"
server_key_iv: "d2f4d5fb7e3242ee626d1dfaa78f4929"
server_key_version: "encryptionkey@0"
I20260620 20:14:43.500296 25375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "e899d4037b454322a9f4dc645c7abbd4"
format_stamp: "Formatted at 2026-06-20 20:14:43 on dist-test-slave-4bwq"
server_key: "6a8cc594f4ea55ca6ca017eec1f3e8d7"
server_key_iv: "d2f4d5fb7e3242ee626d1dfaa78f4929"
server_key_version: "encryptionkey@0"
I20260620 20:14:43.505409 25375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:14:43.508944 25390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:43.510257 25375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:43.510488 25375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "e899d4037b454322a9f4dc645c7abbd4"
format_stamp: "Formatted at 2026-06-20 20:14:43 on dist-test-slave-4bwq"
server_key: "6a8cc594f4ea55ca6ca017eec1f3e8d7"
server_key_iv: "d2f4d5fb7e3242ee626d1dfaa78f4929"
server_key_version: "encryptionkey@0"
I20260620 20:14:43.510716 25375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:43.516429 25375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:43.520364 25375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:43.520663 25375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:43.539062 25375 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33417
I20260620 20:14:43.539101 25441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33417 every 8 connection(s)
I20260620 20:14:43.540468 25375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:43.544106 25442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:43.548339 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 25375
I20260620 20:14:43.548635 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:43.549073 4774 external_mini_cluster.cc:1428] Setting key 40a6efbedec07fe0468a3dc4ebd9c2fd
I20260620 20:14:43.554811 25442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4: Bootstrap starting.
I20260620 20:14:43.556784 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:43.550817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36089 (local address 127.0.0.1:33417)
0620 20:14:43.551543 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:43.551554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:43.551598 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:14:43.553452 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:43.553475 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:43.553820 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:43.554201 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:43.555246 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:43.555253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:43.555265 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:43.555278 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:43.555337 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:43.555356 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:43.555591 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:43.555726 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:14:43.555901 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":325,"threads_started":1}
I20260620 20:14:43.558254 25442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:43.559464 25442 log.cc:824] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:43.560590 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:43.562654 25442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4: No bootstrap required, opened a new log
I20260620 20:14:43.568373 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:43.565794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40037 (local address 127.0.0.1:33417)
0620 20:14:43.566036 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:43.566044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:43.566469 (+ 425us) server_negotiation.cc:407] Connection header received
0620 20:14:43.566599 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:43.566602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:43.566651 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:43.566730 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:43.567702 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:43.567706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:43.567709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:43.567723 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:43.567759 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:43.567772 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:43.567862 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:43.567990 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 20:14:43.568041 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:14:43.572296 25442 raft_consensus.cc:348] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e899d4037b454322a9f4dc645c7abbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33417 } }
I20260620 20:14:43.572664 25442 raft_consensus.cc:374] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:43.572783 25442 raft_consensus.cc:729] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e899d4037b454322a9f4dc645c7abbd4, State: Initialized, Role: FOLLOWER
I20260620 20:14:43.573347 25442 consensus_queue.cc:260] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e899d4037b454322a9f4dc645c7abbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33417 } }
I20260620 20:14:43.573598 25442 raft_consensus.cc:388] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:43.573745 25442 raft_consensus.cc:482] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:43.573912 25442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:43.576320 25442 raft_consensus.cc:504] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e899d4037b454322a9f4dc645c7abbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33417 } }
I20260620 20:14:43.576810 25442 leader_election.cc:302] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e899d4037b454322a9f4dc645c7abbd4; no voters:
I20260620 20:14:43.577342 25442 leader_election.cc:288] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:43.577479 25452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:43.578174 25452 raft_consensus.cc:686] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [term 1 LEADER]: Becoming Leader. State: Replica: e899d4037b454322a9f4dc645c7abbd4, State: Running, Role: LEADER
I20260620 20:14:43.578768 25452 consensus_queue.cc:237] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e899d4037b454322a9f4dc645c7abbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33417 } }
I20260620 20:14:43.579432 25442 sys_catalog.cc:564] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:43.581452 25453 sys_catalog.cc:455] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e899d4037b454322a9f4dc645c7abbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e899d4037b454322a9f4dc645c7abbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33417 } } }
I20260620 20:14:43.581810 25453 sys_catalog.cc:458] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:43.581975 25454 sys_catalog.cc:455] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e899d4037b454322a9f4dc645c7abbd4. Latest consensus state: current_term: 1 leader_uuid: "e899d4037b454322a9f4dc645c7abbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e899d4037b454322a9f4dc645c7abbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33417 } } }
I20260620 20:14:43.582450 25454 sys_catalog.cc:458] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:43.584084 25458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:43.589514 25458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:43.597429 25458 catalog_manager.cc:1347] Generated new cluster ID: fa54593ec9f345c48330fa457581ba82
I20260620 20:14:43.597539 25458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:43.613238 25458 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:43.613698 25458 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:43.619760 25458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e899d4037b454322a9f4dc645c7abbd4: Generated new TSK 0
I20260620 20:14:43.620502 25458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:43.633404 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25375
W20260620 20:14:43.635052 25437 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36089
W20260620 20:14:43.635780 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25375: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:14:43.635941 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 25375
2026-06-20T20:14:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:43 dist-test-slave-4bwq krb5kdc[25473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:43 dist-test-slave-4bwq krb5kdc[25473](info): set up 2 sockets
Jun 20 20:14:43 dist-test-slave-4bwq krb5kdc[25473](info): commencing operation
krb5kdc: starting...
W20260620 20:14:44.743112 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:44 dist-test-slave-4bwq krb5kdc[25473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:44.998163 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:45.332083 25486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:45.332439 25486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:45.332597 25486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:45.340077 25486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:45.340247 25486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:45.340353 25486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:45.340458 25486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:45.340549 25486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:45.350652 25486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:45.352013 25486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:45.353363 25486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:45.365204 25494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:45.365165 25492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:45.365237 25491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:45.366089 25486 server_base.cc:1034] running on GCE node
I20260620 20:14:45.366928 25486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:45.368307 25486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:45.369513 25486 hybrid_clock.cc:648] HybridClock initialized: now 1781986485369462 us; error 35 us; skew 500 ppm
I20260620 20:14:45.369865 25486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:45 dist-test-slave-4bwq krb5kdc[25473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:45.373085 25486 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:45.374552 25486 webserver.cc:460] Webserver started at http://127.0.0.1:38571/ using document root <none> and password file <none>
I20260620 20:14:45.375247 25486 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:45.375371 25486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:45.375681 25486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:45.378621 25486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "2a23749b08864f469167d955e8c068d0"
format_stamp: "Formatted at 2026-06-20 20:14:45 on dist-test-slave-4bwq"
server_key: "0de7e0ce29ffef56fea8fa4867317244"
server_key_iv: "4b960b5fdb42d21ec478d67f33d132eb"
server_key_version: "encryptionkey@0"
I20260620 20:14:45.379367 25486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "2a23749b08864f469167d955e8c068d0"
format_stamp: "Formatted at 2026-06-20 20:14:45 on dist-test-slave-4bwq"
server_key: "0de7e0ce29ffef56fea8fa4867317244"
server_key_iv: "4b960b5fdb42d21ec478d67f33d132eb"
server_key_version: "encryptionkey@0"
I20260620 20:14:45.384341 25486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:14:45.387663 25501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:45.388921 25486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:45.389102 25486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "2a23749b08864f469167d955e8c068d0"
format_stamp: "Formatted at 2026-06-20 20:14:45 on dist-test-slave-4bwq"
server_key: "0de7e0ce29ffef56fea8fa4867317244"
server_key_iv: "4b960b5fdb42d21ec478d67f33d132eb"
server_key_version: "encryptionkey@0"
I20260620 20:14:45.389297 25486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:45.408244 25486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:45.412712 25486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:45.413003 25486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:45.432101 25486 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35527
I20260620 20:14:45.432158 25552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35527 every 8 connection(s)
I20260620 20:14:45.433494 25486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:45.437434 25553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:45.442581 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 25486
I20260620 20:14:45.442765 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:45.443287 4774 external_mini_cluster.cc:1428] Setting key 27cdcae403d5c57cd482d0624d1b586e
I20260620 20:14:45.449728 25553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0: Bootstrap starting.
I20260620 20:14:45.453558 25553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:45.454675 25553 log.cc:824] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:45.457485 25553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0: No bootstrap required, opened a new log
Jun 20 20:14:45 dist-test-slave-4bwq krb5kdc[25473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:14:45.466853 25553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a23749b08864f469167d955e8c068d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260620 20:14:45.467366 25553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:45.467489 25553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a23749b08864f469167d955e8c068d0, State: Initialized, Role: FOLLOWER
I20260620 20:14:45.468096 25553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a23749b08864f469167d955e8c068d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260620 20:14:45.468365 25553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:45.468521 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:45.445213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45463 (local address 127.0.0.1:35527)
0620 20:14:45.446093 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:45.446113 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:45.446170 (+ 57us) server_negotiation.cc:407] Connection header received
0620 20:14:45.448639 (+ 2469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:45.448667 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:45.449176 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:45.449621 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:45.451112 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:45.453207 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:45.454596 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:45.455619 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:45.455760 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:45.460474 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:45.460482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:45.460500 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:45.460540 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:45.463925 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:45.464891 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:45.464898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:45.464904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:45.465092 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:45.465508 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:45.465512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:45.465514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:45.466067 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:45.466304 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:45.466789 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 20:14:45.467198 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":363,"threads_started":1}
I20260620 20:14:45.468583 25553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:45.468824 25553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:45.471900 25553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a23749b08864f469167d955e8c068d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260620 20:14:45.472467 25553 leader_election.cc:302] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a23749b08864f469167d955e8c068d0; no voters:
I20260620 20:14:45.472711 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:45.473150 25553 leader_election.cc:288] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:45.473284 25558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:45.475109 25553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:45.479429 25558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a23749b08864f469167d955e8c068d0, State: Running, Role: LEADER
I20260620 20:14:45.480227 25558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a23749b08864f469167d955e8c068d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260620 20:14:45.483623 25559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a23749b08864f469167d955e8c068d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a23749b08864f469167d955e8c068d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } } }
I20260620 20:14:45.484027 25559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:45.484567 25560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a23749b08864f469167d955e8c068d0. Latest consensus state: current_term: 1 leader_uuid: "2a23749b08864f469167d955e8c068d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a23749b08864f469167d955e8c068d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } } }
I20260620 20:14:45.484845 25560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0 [sys.catalog]: This master's current role is: LEADER
W20260620 20:14:45.498127 25577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:14:45.498345 25577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:14:45.498550 25578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:45.499044 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:45.479970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58771 (local address 127.0.0.1:35527)
0620 20:14:45.480627 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:45.480636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:45.481453 (+ 817us) server_negotiation.cc:407] Connection header received
0620 20:14:45.481808 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:45.481817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:45.481923 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:45.482039 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:45.484341 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:45.485691 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:45.488223 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:45.489175 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:45.489317 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:45.492504 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:45.492515 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:45.492522 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:45.492568 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:45.496035 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:45.496767 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:45.496772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:45.496776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:45.496912 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:45.497311 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:45.497315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:45.497318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:45.497653 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:45.497915 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:45.498256 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 20:14:45.498522 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492}
I20260620 20:14:45.501188 25578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:45.510632 25578 catalog_manager.cc:1347] Generated new cluster ID: a60e1684436d48c09d9d7d59472d0c29
I20260620 20:14:45.510775 25578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:45.520892 25578 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:45.522172 25578 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:45.531795 25578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a23749b08864f469167d955e8c068d0: Generated new TSK 0
I20260620 20:14:45.532855 25578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:45.539417 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25486
I20260620 20:14:45.540467 25548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:45.629189 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 25486
2026-06-20T20:14:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:45 dist-test-slave-4bwq krb5kdc[25585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:45 dist-test-slave-4bwq krb5kdc[25585](info): set up 2 sockets
Jun 20 20:14:45 dist-test-slave-4bwq krb5kdc[25585](info): commencing operation
krb5kdc: starting...
W20260620 20:14:46.739557 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:46 dist-test-slave-4bwq krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:46.991978 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:47.336483 25598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:47.336750 25598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:47.336843 25598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:47.344132 25598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:47.344237 25598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:47.344283 25598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:47.344343 25598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:47.344384 25598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:47.353915 25598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:47.355134 25598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:47.356364 25598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:47.367870 25603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:47.367902 25606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:47.368085 25604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:47.368578 25598 server_base.cc:1034] running on GCE node
I20260620 20:14:47.369347 25598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:47.370687 25598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:47.371882 25598 hybrid_clock.cc:648] HybridClock initialized: now 1781986487371852 us; error 30 us; skew 500 ppm
I20260620 20:14:47.372244 25598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:47 dist-test-slave-4bwq krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:47.375272 25598 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:47.376674 25598 webserver.cc:460] Webserver started at http://127.0.0.1:33729/ using document root <none> and password file <none>
I20260620 20:14:47.377292 25598 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:47.377389 25598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:47.377627 25598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:47.380462 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "cb95b7bf87fb477a8637d5b9be3cbb25"
format_stamp: "Formatted at 2026-06-20 20:14:47 on dist-test-slave-4bwq"
server_key: "73fe2ef8a60a94abb478b71372ab5625"
server_key_iv: "2642f62b586de0a29384511777889165"
server_key_version: "encryptionkey@0"
I20260620 20:14:47.381135 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "cb95b7bf87fb477a8637d5b9be3cbb25"
format_stamp: "Formatted at 2026-06-20 20:14:47 on dist-test-slave-4bwq"
server_key: "73fe2ef8a60a94abb478b71372ab5625"
server_key_iv: "2642f62b586de0a29384511777889165"
server_key_version: "encryptionkey@0"
I20260620 20:14:47.385999 25598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:14:47.389258 25613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:47.390563 25598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:14:47.390743 25598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "cb95b7bf87fb477a8637d5b9be3cbb25"
format_stamp: "Formatted at 2026-06-20 20:14:47 on dist-test-slave-4bwq"
server_key: "73fe2ef8a60a94abb478b71372ab5625"
server_key_iv: "2642f62b586de0a29384511777889165"
server_key_version: "encryptionkey@0"
I20260620 20:14:47.390942 25598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:47.406940 25598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:47.411039 25598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:47.411351 25598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:47.430305 25598 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34127
I20260620 20:14:47.430339 25664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34127 every 8 connection(s)
I20260620 20:14:47.431775 25598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:47.435653 25665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:47.436408 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 25598
I20260620 20:14:47.436638 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:47.437196 4774 external_mini_cluster.cc:1428] Setting key 59d404d28c20be819e529d3958817c0f
I20260620 20:14:47.449548 25665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25: Bootstrap starting.
I20260620 20:14:47.451088 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:47.439036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40733 (local address 127.0.0.1:34127)
0620 20:14:47.439806 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:47.439818 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:47.439860 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:14:47.441923 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:47.441944 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:47.442332 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:47.442759 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:47.443381 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:47.444579 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:47.446654 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:47.447365 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:47.447494 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:47.448745 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:47.448752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:47.448766 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:47.449293 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:47.449370 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:47.449394 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:47.449477 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:47.449783 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:14:47.450075 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":357,"threads_started":1}
I20260620 20:14:47.452823 25665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:47.454022 25665 log.cc:824] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:47.454733 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:47.456806 25665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25: No bootstrap required, opened a new log
I20260620 20:14:47.465224 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:47.459010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48385 (local address 127.0.0.1:34127)
0620 20:14:47.459165 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:47.459169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:47.459688 (+ 519us) server_negotiation.cc:407] Connection header received
0620 20:14:47.459908 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:47.459913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:47.459983 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:47.460100 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:47.460620 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:47.461350 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:47.462697 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:47.463166 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:47.463326 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:47.464488 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:47.464493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:47.464496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:47.464515 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:47.464555 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:47.464570 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:47.464670 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:47.464840 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 20:14:47.464929 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:14:47.465821 25665 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb95b7bf87fb477a8637d5b9be3cbb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } }
I20260620 20:14:47.466274 25665 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:47.466435 25665 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb95b7bf87fb477a8637d5b9be3cbb25, State: Initialized, Role: FOLLOWER
I20260620 20:14:47.467155 25665 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb95b7bf87fb477a8637d5b9be3cbb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } }
I20260620 20:14:47.467411 25665 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:47.467576 25665 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:47.467756 25665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:47.470458 25665 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb95b7bf87fb477a8637d5b9be3cbb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } }
I20260620 20:14:47.471032 25665 leader_election.cc:302] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb95b7bf87fb477a8637d5b9be3cbb25; no voters:
I20260620 20:14:47.471625 25665 leader_election.cc:288] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:47.471771 25675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:47.472469 25675 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [term 1 LEADER]: Becoming Leader. State: Replica: cb95b7bf87fb477a8637d5b9be3cbb25, State: Running, Role: LEADER
I20260620 20:14:47.473037 25675 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb95b7bf87fb477a8637d5b9be3cbb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } }
I20260620 20:14:47.473871 25665 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:47.475865 25677 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb95b7bf87fb477a8637d5b9be3cbb25. Latest consensus state: current_term: 1 leader_uuid: "cb95b7bf87fb477a8637d5b9be3cbb25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb95b7bf87fb477a8637d5b9be3cbb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } } }
I20260620 20:14:47.475989 25676 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb95b7bf87fb477a8637d5b9be3cbb25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb95b7bf87fb477a8637d5b9be3cbb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34127 } } }
I20260620 20:14:47.476329 25677 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:47.476398 25676 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:47.477325 25680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:47.484546 25680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:47.493709 25680 catalog_manager.cc:1347] Generated new cluster ID: 34c7b17156c648bbb1809aee4e03a0fd
I20260620 20:14:47.493853 25680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:47.517637 25680 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:47.519068 25680 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:47.527621 25680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb95b7bf87fb477a8637d5b9be3cbb25: Generated new TSK 0
I20260620 20:14:47.528452 25680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:47.540480 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25598
W20260620 20:14:47.541543 25660 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40733
W20260620 20:14:47.542322 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25598: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:14:47.542557 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 25598
2026-06-20T20:14:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:47 dist-test-slave-4bwq krb5kdc[25696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:47 dist-test-slave-4bwq krb5kdc[25696](info): set up 2 sockets
Jun 20 20:14:47 dist-test-slave-4bwq krb5kdc[25696](info): commencing operation
krb5kdc: starting...
W20260620 20:14:48.639012 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:48 dist-test-slave-4bwq krb5kdc[25696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:48.891733 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:49.227434 25709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:49.227705 25709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:49.227795 25709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:49.235018 25709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:49.235143 25709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:49.235224 25709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:49.235280 25709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:49.235325 25709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:49.245618 25709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:49.246934 25709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:49.248252 25709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:49.260780 25717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:49.260809 25714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:49.260735 25715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:49.262024 25709 server_base.cc:1034] running on GCE node
I20260620 20:14:49.262948 25709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:49.264464 25709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:49.265707 25709 hybrid_clock.cc:648] HybridClock initialized: now 1781986489265655 us; error 66 us; skew 500 ppm
I20260620 20:14:49.266080 25709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:49 dist-test-slave-4bwq krb5kdc[25696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:49.273228 25709 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:49.274837 25709 webserver.cc:460] Webserver started at http://127.0.0.1:46089/ using document root <none> and password file <none>
I20260620 20:14:49.275585 25709 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:49.275735 25709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:49.276050 25709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:49.279237 25709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "38a7f18bf7d048eabd12d41c82b7d31e"
format_stamp: "Formatted at 2026-06-20 20:14:49 on dist-test-slave-4bwq"
server_key: "e236737f316fdc1be48b32045fbba03c"
server_key_iv: "1805f94412fc883bf5ab61f25fec3d42"
server_key_version: "encryptionkey@0"
I20260620 20:14:49.280051 25709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "38a7f18bf7d048eabd12d41c82b7d31e"
format_stamp: "Formatted at 2026-06-20 20:14:49 on dist-test-slave-4bwq"
server_key: "e236737f316fdc1be48b32045fbba03c"
server_key_iv: "1805f94412fc883bf5ab61f25fec3d42"
server_key_version: "encryptionkey@0"
I20260620 20:14:49.285389 25709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 20:14:49.288961 25724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:49.290321 25709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:49.290548 25709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "38a7f18bf7d048eabd12d41c82b7d31e"
format_stamp: "Formatted at 2026-06-20 20:14:49 on dist-test-slave-4bwq"
server_key: "e236737f316fdc1be48b32045fbba03c"
server_key_iv: "1805f94412fc883bf5ab61f25fec3d42"
server_key_version: "encryptionkey@0"
I20260620 20:14:49.290763 25709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:49.296476 25709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:49.300848 25709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:49.301175 25709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:49.320719 25709 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43001
I20260620 20:14:49.320760 25775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43001 every 8 connection(s)
I20260620 20:14:49.322180 25709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:49.325947 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:49.326474 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 25709
I20260620 20:14:49.326788 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:49.327430 4774 external_mini_cluster.cc:1428] Setting key c81c59551b45f631cea1182e75918a16
I20260620 20:14:49.336107 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:49.329340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53359 (local address 127.0.0.1:43001)
0620 20:14:49.330174 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:49.330196 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:49.330239 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:14:49.332625 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:49.332653 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:49.333008 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:49.333465 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:49.334552 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:49.334559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:49.334571 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:49.334587 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:49.334644 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:49.334667 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:49.334922 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:49.335053 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 20:14:49.335230 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":360,"threads_started":1}
I20260620 20:14:49.339767 25776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e: Bootstrap starting.
I20260620 20:14:49.339999 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:49.343773 25776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:49.345233 25776 log.cc:824] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:49.348095 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:49.345434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58897 (local address 127.0.0.1:43001)
0620 20:14:49.345649 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:49.345655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:49.346038 (+ 383us) server_negotiation.cc:407] Connection header received
0620 20:14:49.346253 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:49.346262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:49.346327 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:49.346457 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:49.347437 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:49.347443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:49.347448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:49.347466 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:49.347513 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:49.347529 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:49.347650 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:49.347765 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:14:49.347827 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:14:49.348277 25776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e: No bootstrap required, opened a new log
I20260620 20:14:49.356328 25776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a7f18bf7d048eabd12d41c82b7d31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260620 20:14:49.356761 25776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:49.356884 25776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a7f18bf7d048eabd12d41c82b7d31e, State: Initialized, Role: FOLLOWER
I20260620 20:14:49.357467 25776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a7f18bf7d048eabd12d41c82b7d31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260620 20:14:49.357702 25776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:49.357856 25776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:49.358026 25776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:49.360425 25776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a7f18bf7d048eabd12d41c82b7d31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260620 20:14:49.361050 25776 leader_election.cc:302] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38a7f18bf7d048eabd12d41c82b7d31e; no voters:
I20260620 20:14:49.361662 25776 leader_election.cc:288] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:49.361809 25786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:49.362514 25786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [term 1 LEADER]: Becoming Leader. State: Replica: 38a7f18bf7d048eabd12d41c82b7d31e, State: Running, Role: LEADER
I20260620 20:14:49.363067 25786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a7f18bf7d048eabd12d41c82b7d31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260620 20:14:49.363806 25776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:49.365864 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38a7f18bf7d048eabd12d41c82b7d31e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a7f18bf7d048eabd12d41c82b7d31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } } }
I20260620 20:14:49.366271 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:49.366782 25788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a7f18bf7d048eabd12d41c82b7d31e. Latest consensus state: current_term: 1 leader_uuid: "38a7f18bf7d048eabd12d41c82b7d31e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a7f18bf7d048eabd12d41c82b7d31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } } }
I20260620 20:14:49.367105 25788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:49.368744 25792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:49.374231 25792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:49.382546 25792 catalog_manager.cc:1347] Generated new cluster ID: f15a5db8ef2445278c2345a195e73954
I20260620 20:14:49.382666 25792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:49.396814 25792 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:49.397255 25792 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:49.403749 25792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38a7f18bf7d048eabd12d41c82b7d31e: Generated new TSK 0
I20260620 20:14:49.404546 25792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:49.417316 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25709
W20260620 20:14:49.418473 25771 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53359
W20260620 20:14:49.419286 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25709: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:14:49.419445 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 25709
2026-06-20T20:14:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:49 dist-test-slave-4bwq krb5kdc[25807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:49 dist-test-slave-4bwq krb5kdc[25807](info): set up 2 sockets
Jun 20 20:14:49 dist-test-slave-4bwq krb5kdc[25807](info): commencing operation
krb5kdc: starting...
W20260620 20:14:50.627274 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:50 dist-test-slave-4bwq krb5kdc[25807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:50.879537 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:51.210539 25820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:51.210894 25820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:51.211061 25820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:51.218434 25820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:51.218606 25820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:51.218708 25820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:51.218813 25820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:51.218899 25820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:51.228873 25820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:51.230296 25820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:51.231655 25820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:51.243680 25826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:51.243814 25828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:51.243933 25825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:51.244776 25820 server_base.cc:1034] running on GCE node
I20260620 20:14:51.245659 25820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:51.247190 25820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:51.248399 25820 hybrid_clock.cc:648] HybridClock initialized: now 1781986491248334 us; error 54 us; skew 500 ppm
I20260620 20:14:51.248770 25820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:51 dist-test-slave-4bwq krb5kdc[25807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:51.252060 25820 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:51.253665 25820 webserver.cc:460] Webserver started at http://127.0.0.1:39493/ using document root <none> and password file <none>
I20260620 20:14:51.254487 25820 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:51.254630 25820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:51.254940 25820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:51.258003 25820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "d1bda36ba4344ba493e730f65452dd17"
format_stamp: "Formatted at 2026-06-20 20:14:51 on dist-test-slave-4bwq"
server_key: "92d7ca576358ace25d444e628954152a"
server_key_iv: "07ad9b10344b4bfc7035b216693fa2a6"
server_key_version: "encryptionkey@0"
I20260620 20:14:51.258816 25820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "d1bda36ba4344ba493e730f65452dd17"
format_stamp: "Formatted at 2026-06-20 20:14:51 on dist-test-slave-4bwq"
server_key: "92d7ca576358ace25d444e628954152a"
server_key_iv: "07ad9b10344b4bfc7035b216693fa2a6"
server_key_version: "encryptionkey@0"
I20260620 20:14:51.263823 25820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:14:51.267131 25835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:51.268522 25820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:51.268720 25820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "d1bda36ba4344ba493e730f65452dd17"
format_stamp: "Formatted at 2026-06-20 20:14:51 on dist-test-slave-4bwq"
server_key: "92d7ca576358ace25d444e628954152a"
server_key_iv: "07ad9b10344b4bfc7035b216693fa2a6"
server_key_version: "encryptionkey@0"
I20260620 20:14:51.268940 25820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:51.292423 25820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:51.297319 25820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:51.297657 25820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:51.318279 25820 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39435
I20260620 20:14:51.318286 25886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39435 every 8 connection(s)
I20260620 20:14:51.319844 25820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:51.323195 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 25820
I20260620 20:14:51.323421 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:51.323856 25887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:51.323921 4774 external_mini_cluster.cc:1428] Setting key b8fde07d497286c8776e6448a37e3f00
I20260620 20:14:51.338269 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17: Bootstrap starting.
Jun 20 20:14:51 dist-test-slave-4bwq krb5kdc[25807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:14:51.342075 25887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:51.343537 25887 log.cc:824] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:51.346935 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17: No bootstrap required, opened a new log
I20260620 20:14:51.349383 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:51.325736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:50435 (local address 127.0.0.1:39435)
0620 20:14:51.326513 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:51.326529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:51.326596 (+ 67us) server_negotiation.cc:407] Connection header received
0620 20:14:51.328872 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:51.328902 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:51.329448 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:51.329916 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:51.331516 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:51.333426 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:51.334855 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:51.335976 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:51.336176 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:51.340562 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:51.340572 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:51.340597 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:51.340644 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:51.344705 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:51.345617 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:51.345625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:51.345633 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:51.345840 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:51.346241 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:51.346246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:51.346249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:51.346951 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:51.347293 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:51.347713 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 20:14:51.348049 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":311,"threads_started":1}
I20260620 20:14:51.353597 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:51.356698 25887 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bda36ba4344ba493e730f65452dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39435 } }
I20260620 20:14:51.357219 25887 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:51.357338 25887 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bda36ba4344ba493e730f65452dd17, State: Initialized, Role: FOLLOWER
I20260620 20:14:51.358080 25887 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bda36ba4344ba493e730f65452dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39435 } }
I20260620 20:14:51.358410 25887 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:51.358558 25887 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:51.358767 25887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:51.361719 25887 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bda36ba4344ba493e730f65452dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39435 } }
I20260620 20:14:51.362306 25887 leader_election.cc:302] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1bda36ba4344ba493e730f65452dd17; no voters:
I20260620 20:14:51.363005 25887 leader_election.cc:288] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:51.363140 25897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:51.365242 25887 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:51.365254 25897 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [term 1 LEADER]: Becoming Leader. State: Replica: d1bda36ba4344ba493e730f65452dd17, State: Running, Role: LEADER
I20260620 20:14:51.365964 25897 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bda36ba4344ba493e730f65452dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39435 } }
I20260620 20:14:51.368733 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1bda36ba4344ba493e730f65452dd17. Latest consensus state: current_term: 1 leader_uuid: "d1bda36ba4344ba493e730f65452dd17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bda36ba4344ba493e730f65452dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39435 } } }
I20260620 20:14:51.369122 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:51.370654 25898 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1bda36ba4344ba493e730f65452dd17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bda36ba4344ba493e730f65452dd17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39435 } } }
I20260620 20:14:51.371508 25898 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:51.377190 25902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:51.380116 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:51.359108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:36873 (local address 127.0.0.1:39435)
0620 20:14:51.359279 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:51.359286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:51.359729 (+ 443us) server_negotiation.cc:407] Connection header received
0620 20:14:51.359878 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:51.359887 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:51.359969 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:51.360090 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:51.361506 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:51.362688 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:51.364863 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:51.365661 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:51.365814 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:51.369698 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:51.369710 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:51.369717 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:51.369771 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:51.374705 (+ 4934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:51.376172 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:51.376180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:51.376186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:51.376373 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:51.376949 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:51.376957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:51.376967 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:51.377498 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:51.377868 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:51.379258 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0620 20:14:51.379539 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:14:51.382651 25902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:51.395262 25902 catalog_manager.cc:1347] Generated new cluster ID: e4da9381ccff43be808a788f11b16707
I20260620 20:14:51.395756 25902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:51.409088 25902 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:51.410542 25902 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:51.424630 25902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1bda36ba4344ba493e730f65452dd17: Generated new TSK 0
I20260620 20:14:51.425733 25902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:51.441707 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25820
I20260620 20:14:51.442927 25882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:51.537418 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 25820
W20260620 20:14:51.554670 25805 connection.cc:535] client connection to 127.0.0.1:39435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39435): Connection reset by peer (error 104)
2026-06-20T20:14:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:51 dist-test-slave-4bwq krb5kdc[25919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:51 dist-test-slave-4bwq krb5kdc[25919](info): set up 2 sockets
Jun 20 20:14:51 dist-test-slave-4bwq krb5kdc[25919](info): commencing operation
krb5kdc: starting...
W20260620 20:14:52.643039 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:52 dist-test-slave-4bwq krb5kdc[25919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:52.893049 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:53.226083 25932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:53.226372 25932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:53.226549 25932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:53.233822 25932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:53.233940 25932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:53.234042 25932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:53.234123 25932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:53.234169 25932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:53.244242 25932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:53.245579 25932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:53.246934 25932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:53.258741 25937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:53.258790 25938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:53.258822 25940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:53.259586 25932 server_base.cc:1034] running on GCE node
I20260620 20:14:53.260372 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:53.261773 25932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:53.262987 25932 hybrid_clock.cc:648] HybridClock initialized: now 1781986493262946 us; error 38 us; skew 500 ppm
I20260620 20:14:53.263338 25932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:53 dist-test-slave-4bwq krb5kdc[25919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:53.266451 25932 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:53.267967 25932 webserver.cc:460] Webserver started at http://127.0.0.1:41825/ using document root <none> and password file <none>
I20260620 20:14:53.268657 25932 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:53.268795 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:53.269088 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:53.272080 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "de10431260584f92aa42e3b77a158384"
format_stamp: "Formatted at 2026-06-20 20:14:53 on dist-test-slave-4bwq"
server_key: "c9ac49555382c771c3cf5535b645036f"
server_key_iv: "f13155c1b64bddfb15bf6f68c879ad38"
server_key_version: "encryptionkey@0"
I20260620 20:14:53.272845 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "de10431260584f92aa42e3b77a158384"
format_stamp: "Formatted at 2026-06-20 20:14:53 on dist-test-slave-4bwq"
server_key: "c9ac49555382c771c3cf5535b645036f"
server_key_iv: "f13155c1b64bddfb15bf6f68c879ad38"
server_key_version: "encryptionkey@0"
I20260620 20:14:53.277833 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:14:53.281224 25947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:53.282553 25932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:53.282733 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "de10431260584f92aa42e3b77a158384"
format_stamp: "Formatted at 2026-06-20 20:14:53 on dist-test-slave-4bwq"
server_key: "c9ac49555382c771c3cf5535b645036f"
server_key_iv: "f13155c1b64bddfb15bf6f68c879ad38"
server_key_version: "encryptionkey@0"
I20260620 20:14:53.282927 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:53.301708 25932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:53.305747 25932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:53.306046 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:53.324740 25932 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45999
I20260620 20:14:53.324777 25998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45999 every 8 connection(s)
I20260620 20:14:53.326104 25932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:53.327456 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 25932
I20260620 20:14:53.327656 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:53.328215 4774 external_mini_cluster.cc:1428] Setting key e386637f79a8ed5be9e57f1f9c6f2945
I20260620 20:14:53.330778 25999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:53.340776 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:53.330002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:35863 (local address 127.0.0.1:45999)
0620 20:14:53.330778 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:53.330790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:53.330849 (+ 59us) server_negotiation.cc:407] Connection header received
0620 20:14:53.332462 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:53.332483 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:53.332845 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:53.333241 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:53.333780 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:53.335174 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:53.336479 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:53.337130 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:53.337320 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:53.338648 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:53.338656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:53.338671 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:53.339213 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:53.339288 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:53.339314 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:53.339478 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:53.339742 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 20:14:53.339891 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":381,"threads_started":1}
I20260620 20:14:53.341639 25999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384: Bootstrap starting.
I20260620 20:14:53.344326 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:53.344836 25999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:53.345764 25999 log.cc:824] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:53.348529 25999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384: No bootstrap required, opened a new log
I20260620 20:14:53.355070 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:53.348680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:58123 (local address 127.0.0.1:45999)
0620 20:14:53.348856 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:53.348861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:53.349241 (+ 380us) server_negotiation.cc:407] Connection header received
0620 20:14:53.349383 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:53.349389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:53.349467 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:53.349566 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:53.350009 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:53.351227 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:53.352439 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:53.353104 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:53.353228 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:53.354288 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:53.354294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:53.354299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:53.354320 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:53.354413 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:53.354434 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:53.354549 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:53.354680 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 20:14:53.354736 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:14:53.358047 25999 raft_consensus.cc:348] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de10431260584f92aa42e3b77a158384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260620 20:14:53.358462 25999 raft_consensus.cc:374] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:53.358546 25999 raft_consensus.cc:729] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de10431260584f92aa42e3b77a158384, State: Initialized, Role: FOLLOWER
I20260620 20:14:53.359040 25999 consensus_queue.cc:260] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de10431260584f92aa42e3b77a158384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260620 20:14:53.359244 25999 raft_consensus.cc:388] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:53.359338 25999 raft_consensus.cc:482] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:53.359467 25999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:53.361584 25999 raft_consensus.cc:504] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de10431260584f92aa42e3b77a158384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260620 20:14:53.362073 25999 leader_election.cc:302] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de10431260584f92aa42e3b77a158384; no voters:
I20260620 20:14:53.362775 25999 leader_election.cc:288] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:53.362874 26009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:53.363595 26009 raft_consensus.cc:686] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [term 1 LEADER]: Becoming Leader. State: Replica: de10431260584f92aa42e3b77a158384, State: Running, Role: LEADER
I20260620 20:14:53.364177 26009 consensus_queue.cc:237] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de10431260584f92aa42e3b77a158384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260620 20:14:53.364876 25999 sys_catalog.cc:564] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:53.366780 26010 sys_catalog.cc:455] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de10431260584f92aa42e3b77a158384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de10431260584f92aa42e3b77a158384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } } }
I20260620 20:14:53.367097 26010 sys_catalog.cc:458] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:53.367933 26015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:53.367776 26011 sys_catalog.cc:455] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de10431260584f92aa42e3b77a158384. Latest consensus state: current_term: 1 leader_uuid: "de10431260584f92aa42e3b77a158384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de10431260584f92aa42e3b77a158384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } } }
I20260620 20:14:53.368124 26011 sys_catalog.cc:458] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:53.373085 26015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:53.381820 26015 catalog_manager.cc:1347] Generated new cluster ID: ba8e7ffccd584b50868994b6a8dc4738
I20260620 20:14:53.381928 26015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:53.411773 26015 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:53.413188 26015 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:53.421146 26015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de10431260584f92aa42e3b77a158384: Generated new TSK 0
I20260620 20:14:53.421984 26015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:53.429890 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25932
W20260620 20:14:53.430987 25994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:35863
W20260620 20:14:53.431742 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 25932: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:14:53.431902 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 25932
W20260620 20:14:53.448786 25917 connection.cc:535] client connection to 127.0.0.1:45999 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45999): Connection reset by peer (error 104)
2026-06-20T20:14:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:53 dist-test-slave-4bwq krb5kdc[26030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:53 dist-test-slave-4bwq krb5kdc[26030](info): set up 2 sockets
Jun 20 20:14:53 dist-test-slave-4bwq krb5kdc[26030](info): commencing operation
krb5kdc: starting...
W20260620 20:14:54.527053 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:54 dist-test-slave-4bwq krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:54.783794 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:55.117110 26043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:55.117398 26043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:55.117560 26043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:55.124773 26043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:55.124892 26043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:55.125022 26043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:55.125130 26043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:55.125218 26043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:55.135403 26043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:55.136806 26043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:55.138165 26043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:55.150244 26049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:55.150282 26048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:55.150311 26051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:55.151432 26043 server_base.cc:1034] running on GCE node
I20260620 20:14:55.152400 26043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:55.153965 26043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:55.155177 26043 hybrid_clock.cc:648] HybridClock initialized: now 1781986495155115 us; error 52 us; skew 500 ppm
I20260620 20:14:55.155498 26043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:55 dist-test-slave-4bwq krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:55.162935 26043 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:55.164515 26043 webserver.cc:460] Webserver started at http://127.0.0.1:40039/ using document root <none> and password file <none>
I20260620 20:14:55.165233 26043 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:55.165378 26043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:55.165679 26043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:55.168857 26043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "e6ca7fa044de49218b230116fce94aba"
format_stamp: "Formatted at 2026-06-20 20:14:55 on dist-test-slave-4bwq"
server_key: "9c6aa3ebf5976fb845dcb18e65569591"
server_key_iv: "c95c5dc26f275735b5b26740766e1e77"
server_key_version: "encryptionkey@0"
I20260620 20:14:55.169618 26043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "e6ca7fa044de49218b230116fce94aba"
format_stamp: "Formatted at 2026-06-20 20:14:55 on dist-test-slave-4bwq"
server_key: "9c6aa3ebf5976fb845dcb18e65569591"
server_key_iv: "c95c5dc26f275735b5b26740766e1e77"
server_key_version: "encryptionkey@0"
I20260620 20:14:55.174758 26043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:14:55.178172 26058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:55.179446 26043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:55.179637 26043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "e6ca7fa044de49218b230116fce94aba"
format_stamp: "Formatted at 2026-06-20 20:14:55 on dist-test-slave-4bwq"
server_key: "9c6aa3ebf5976fb845dcb18e65569591"
server_key_iv: "c95c5dc26f275735b5b26740766e1e77"
server_key_version: "encryptionkey@0"
I20260620 20:14:55.179893 26043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:55.185402 26043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:55.189209 26043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:55.189494 26043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:55.207710 26043 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40973
I20260620 20:14:55.207788 26109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40973 every 8 connection(s)
I20260620 20:14:55.209067 26043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:55.212783 26110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:55.218060 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 26043
I20260620 20:14:55.218254 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:55.218739 4774 external_mini_cluster.cc:1428] Setting key b64089c1dfbd45926ff69ba44f7cbfbb
I20260620 20:14:55.223559 26110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba: Bootstrap starting.
I20260620 20:14:55.226637 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:55.220612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:40473 (local address 127.0.0.1:40973)
0620 20:14:55.221484 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:55.221496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:55.221538 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:14:55.223263 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:55.223287 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:55.223622 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:55.223988 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:55.225044 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:55.225050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:55.225062 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:55.225075 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:55.225161 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:55.225182 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:55.225361 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:55.225464 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 20:14:55.225677 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":395,"threads_started":1}
I20260620 20:14:55.227279 26110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:55.228351 26110 log.cc:824] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:55.230304 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:55.231299 26110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba: No bootstrap required, opened a new log
I20260620 20:14:55.237236 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:55.234720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:37549 (local address 127.0.0.1:40973)
0620 20:14:55.234956 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:55.234962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:55.235303 (+ 341us) server_negotiation.cc:407] Connection header received
0620 20:14:55.235468 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:55.235471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:55.235521 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:55.235606 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:55.236554 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:55.236558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:55.236561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:55.236579 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:55.236617 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:55.236632 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:55.236738 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:55.236853 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:14:55.236906 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:14:55.240238 26110 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ca7fa044de49218b230116fce94aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260620 20:14:55.240645 26110 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:55.240756 26110 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ca7fa044de49218b230116fce94aba, State: Initialized, Role: FOLLOWER
I20260620 20:14:55.241331 26110 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ca7fa044de49218b230116fce94aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260620 20:14:55.241535 26110 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:55.241637 26110 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:55.241760 26110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:55.243992 26110 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ca7fa044de49218b230116fce94aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260620 20:14:55.244446 26110 leader_election.cc:302] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6ca7fa044de49218b230116fce94aba; no voters:
I20260620 20:14:55.244992 26110 leader_election.cc:288] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:55.245142 26120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:55.245868 26120 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [term 1 LEADER]: Becoming Leader. State: Replica: e6ca7fa044de49218b230116fce94aba, State: Running, Role: LEADER
I20260620 20:14:55.246547 26120 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ca7fa044de49218b230116fce94aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260620 20:14:55.247334 26110 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:55.249814 26121 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6ca7fa044de49218b230116fce94aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ca7fa044de49218b230116fce94aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260620 20:14:55.249694 26122 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6ca7fa044de49218b230116fce94aba. Latest consensus state: current_term: 1 leader_uuid: "e6ca7fa044de49218b230116fce94aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ca7fa044de49218b230116fce94aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260620 20:14:55.250195 26121 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:55.250207 26122 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:55.252436 26128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:55.257298 26128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:55.266921 26128 catalog_manager.cc:1347] Generated new cluster ID: f8819764b3914cd39236832d0e82c4d5
I20260620 20:14:55.267057 26128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:55.298364 26128 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:55.298837 26128 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:55.316491 26128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6ca7fa044de49218b230116fce94aba: Generated new TSK 0
I20260620 20:14:55.317382 26128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:55.332979 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26043
W20260620 20:14:55.334136 26105 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:40473
W20260620 20:14:55.334941 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26043: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:14:55.335098 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 26043
2026-06-20T20:14:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1899 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:55 dist-test-slave-4bwq krb5kdc[26141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:55 dist-test-slave-4bwq krb5kdc[26141](info): set up 2 sockets
Jun 20 20:14:55 dist-test-slave-4bwq krb5kdc[26141](info): commencing operation
krb5kdc: starting...
W20260620 20:14:56.446991 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:56 dist-test-slave-4bwq krb5kdc[26141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:56.701290 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:57.033218 26154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:57.033636 26154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:57.033850 26154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:57.041451 26154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:57.041576 26154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:57.041628 26154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:57.041689 26154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:57.041731 26154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:57.052068 26154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:57.053295 26154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:57.054590 26154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:57.066309 26162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:57.066365 26160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:57.066507 26159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:57.067284 26154 server_base.cc:1034] running on GCE node
I20260620 20:14:57.068054 26154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:57.069465 26154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:57.070683 26154 hybrid_clock.cc:648] HybridClock initialized: now 1781986497070642 us; error 56 us; skew 500 ppm
I20260620 20:14:57.071014 26154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:57 dist-test-slave-4bwq krb5kdc[26141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:57.074014 26154 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:57.075574 26154 webserver.cc:460] Webserver started at http://127.0.0.1:43063/ using document root <none> and password file <none>
I20260620 20:14:57.076267 26154 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:57.076406 26154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:57.076709 26154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:57.079694 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "7bf8892b9d684cda8b91b4ec14a33028"
format_stamp: "Formatted at 2026-06-20 20:14:57 on dist-test-slave-4bwq"
server_key: "8db1826fab402275f1ddc0bcb03b1e4c"
server_key_iv: "d5a609a24ae245f874dbd682e5181a4a"
server_key_version: "encryptionkey@0"
I20260620 20:14:57.080457 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "7bf8892b9d684cda8b91b4ec14a33028"
format_stamp: "Formatted at 2026-06-20 20:14:57 on dist-test-slave-4bwq"
server_key: "8db1826fab402275f1ddc0bcb03b1e4c"
server_key_iv: "d5a609a24ae245f874dbd682e5181a4a"
server_key_version: "encryptionkey@0"
I20260620 20:14:57.085860 26154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:14:57.089884 26169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:57.091518 26154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:57.091869 26154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "7bf8892b9d684cda8b91b4ec14a33028"
format_stamp: "Formatted at 2026-06-20 20:14:57 on dist-test-slave-4bwq"
server_key: "8db1826fab402275f1ddc0bcb03b1e4c"
server_key_iv: "d5a609a24ae245f874dbd682e5181a4a"
server_key_version: "encryptionkey@0"
I20260620 20:14:57.092262 26154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:57.118589 26154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:57.122896 26154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:57.123196 26154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:57.142161 26154 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40865
I20260620 20:14:57.142202 26220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40865 every 8 connection(s)
I20260620 20:14:57.143544 26154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:57.146255 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 26154
I20260620 20:14:57.146471 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:57.147028 4774 external_mini_cluster.cc:1428] Setting key a79ba845816a085fdbf7ea969a113466
I20260620 20:14:57.147734 26221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 20:14:57 dist-test-slave-4bwq krb5kdc[26141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:14:57.162212 26221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028: Bootstrap starting.
I20260620 20:14:57.165555 26221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:57.166615 26221 log.cc:824] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:57.169350 26221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028: No bootstrap required, opened a new log
I20260620 20:14:57.169554 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:57.148773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:39953 (local address 127.0.0.1:40865)
0620 20:14:57.149559 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:57.149570 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:57.149614 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:14:57.151177 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:57.151197 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:57.151576 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:57.151901 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:57.153611 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:57.155010 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:57.156770 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:57.157494 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:57.157622 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:57.162490 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:57.162504 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:57.162525 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:57.162569 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:57.165432 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:57.166239 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:57.166245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:57.166251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:57.166513 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:57.167020 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:57.167025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:57.167029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:57.167513 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:57.167843 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:57.168337 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 20:14:57.168628 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":385,"threads_started":1}
I20260620 20:14:57.173130 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:57.178035 26221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf8892b9d684cda8b91b4ec14a33028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40865 } }
I20260620 20:14:57.178519 26221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:57.178640 26221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf8892b9d684cda8b91b4ec14a33028, State: Initialized, Role: FOLLOWER
I20260620 20:14:57.179243 26221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf8892b9d684cda8b91b4ec14a33028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40865 } }
I20260620 20:14:57.179467 26221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:57.179622 26221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:57.179778 26221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:57.182709 26221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf8892b9d684cda8b91b4ec14a33028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40865 } }
I20260620 20:14:57.183219 26221 leader_election.cc:302] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bf8892b9d684cda8b91b4ec14a33028; no voters:
I20260620 20:14:57.183866 26221 leader_election.cc:288] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:57.183974 26231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:57.186735 26221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:57.187021 26231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf8892b9d684cda8b91b4ec14a33028, State: Running, Role: LEADER
I20260620 20:14:57.187633 26231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf8892b9d684cda8b91b4ec14a33028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40865 } }
I20260620 20:14:57.193174 26232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bf8892b9d684cda8b91b4ec14a33028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf8892b9d684cda8b91b4ec14a33028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40865 } } }
I20260620 20:14:57.193343 26233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bf8892b9d684cda8b91b4ec14a33028. Latest consensus state: current_term: 1 leader_uuid: "7bf8892b9d684cda8b91b4ec14a33028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf8892b9d684cda8b91b4ec14a33028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40865 } } }
I20260620 20:14:57.193775 26233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:57.194250 26232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:57.197214 26239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:57.197329 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:57.177687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:45041 (local address 127.0.0.1:40865)
0620 20:14:57.177834 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:57.177838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:57.178317 (+ 479us) server_negotiation.cc:407] Connection header received
0620 20:14:57.178562 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:57.178570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:57.178671 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:57.178781 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:57.180411 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:57.181639 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:57.183014 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:57.183547 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:57.183705 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:57.186947 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:57.186954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:57.186958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:57.187021 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:57.191535 (+ 4514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:57.192819 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:57.192827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:57.192833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:57.193046 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:57.193757 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:57.193766 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:57.193772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:57.194315 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:57.194877 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:57.196495 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0620 20:14:57.196737 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:14:57.202618 26239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:57.211956 26239 catalog_manager.cc:1347] Generated new cluster ID: 36091fbb66b84732aa61d4c5d2bd6fc1
I20260620 20:14:57.212090 26239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:57.222009 26239 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:57.223870 26239 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:57.240536 26239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bf8892b9d684cda8b91b4ec14a33028: Generated new TSK 0
I20260620 20:14:57.241571 26239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:57.256060 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26154
I20260620 20:14:57.257195 26216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:14:57.347035 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 26154
W20260620 20:14:57.363659 26139 connection.cc:535] client connection to 127.0.0.1:40865 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40865): Connection reset by peer (error 104)
2026-06-20T20:14:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:57 dist-test-slave-4bwq krb5kdc[26253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:57 dist-test-slave-4bwq krb5kdc[26253](info): set up 2 sockets
Jun 20 20:14:57 dist-test-slave-4bwq krb5kdc[26253](info): commencing operation
krb5kdc: starting...
W20260620 20:14:58.443050 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:14:58 dist-test-slave-4bwq krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:14:58.711031 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:59.053982 26266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:59.054364 26266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:59.054551 26266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:59.062076 26266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:59.062269 26266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:59.062381 26266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:59.062505 26266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:59.062594 26266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:59.073024 26266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:14:59.074525 26266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:59.075940 26266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:59.088563 26272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:59.088627 26274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:59.088533 26271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:59.089541 26266 server_base.cc:1034] running on GCE node
I20260620 20:14:59.090432 26266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:59.092135 26266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:59.093393 26266 hybrid_clock.cc:648] HybridClock initialized: now 1781986499093349 us; error 54 us; skew 500 ppm
I20260620 20:14:59.093793 26266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:14:59 dist-test-slave-4bwq krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:59.097131 26266 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:14:59.098866 26266 webserver.cc:460] Webserver started at http://127.0.0.1:33243/ using document root <none> and password file <none>
I20260620 20:14:59.099586 26266 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:59.099735 26266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:59.100049 26266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:59.103163 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "70da9b84e91d42d4b330b097b51a5df5"
format_stamp: "Formatted at 2026-06-20 20:14:59 on dist-test-slave-4bwq"
server_key: "44361d4661842d2fbccb668da1aa68f7"
server_key_iv: "0a1dc9e5060803a9b33dd26409c59bcd"
server_key_version: "encryptionkey@0"
I20260620 20:14:59.103960 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "70da9b84e91d42d4b330b097b51a5df5"
format_stamp: "Formatted at 2026-06-20 20:14:59 on dist-test-slave-4bwq"
server_key: "44361d4661842d2fbccb668da1aa68f7"
server_key_iv: "0a1dc9e5060803a9b33dd26409c59bcd"
server_key_version: "encryptionkey@0"
I20260620 20:14:59.109565 26266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 20:14:59.113229 26281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:59.114831 26266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:59.115024 26266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "70da9b84e91d42d4b330b097b51a5df5"
format_stamp: "Formatted at 2026-06-20 20:14:59 on dist-test-slave-4bwq"
server_key: "44361d4661842d2fbccb668da1aa68f7"
server_key_iv: "0a1dc9e5060803a9b33dd26409c59bcd"
server_key_version: "encryptionkey@0"
I20260620 20:14:59.115231 26266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:59.145475 26266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:59.150276 26266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:59.150663 26266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:59.170909 26266 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42499
I20260620 20:14:59.170940 26332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42499 every 8 connection(s)
I20260620 20:14:59.172371 26266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:59.176131 26333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:59.178377 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 26266
I20260620 20:14:59.178598 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:14:59.179148 4774 external_mini_cluster.cc:1428] Setting key 6e1c376c4bae070596e14ca78b8042dd
I20260620 20:14:59.188580 26333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5: Bootstrap starting.
I20260620 20:14:59.192219 26333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:59.193179 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:59.181037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:54063 (local address 127.0.0.1:42499)
0620 20:14:59.182021 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:59.182037 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:59.182091 (+ 54us) server_negotiation.cc:407] Connection header received
0620 20:14:59.183974 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:59.183999 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:59.184436 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:59.184780 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:59.185360 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:59.186925 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:59.188800 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:59.189465 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:59.189594 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:59.190937 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:59.190944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:59.190958 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:59.191477 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:59.191548 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:59.191572 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:59.191690 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:59.191951 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 20:14:59.192189 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":478,"threads_started":1}
I20260620 20:14:59.193315 26333 log.cc:824] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:59.196316 26333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5: No bootstrap required, opened a new log
I20260620 20:14:59.196935 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:14:59.204352 26333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70da9b84e91d42d4b330b097b51a5df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } }
I20260620 20:14:59.204718 26333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:59.204824 26333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70da9b84e91d42d4b330b097b51a5df5, State: Initialized, Role: FOLLOWER
I20260620 20:14:59.205338 26333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70da9b84e91d42d4b330b097b51a5df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } }
I20260620 20:14:59.205610 26333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:59.205742 26333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:59.205912 26333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:59.208240 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:59.201219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:55665 (local address 127.0.0.1:42499)
0620 20:14:59.201387 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:59.201393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:59.201871 (+ 478us) server_negotiation.cc:407] Connection header received
0620 20:14:59.202010 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:59.202016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:59.202101 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:59.202213 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:59.202847 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:59.203985 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:59.205507 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:59.206240 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:59.206416 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:59.207486 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:59.207492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:59.207497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:59.207520 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:59.207577 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:59.207596 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:59.207725 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:59.207878 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 20:14:59.207932 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:14:59.208783 26333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70da9b84e91d42d4b330b097b51a5df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } }
I20260620 20:14:59.209263 26333 leader_election.cc:302] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70da9b84e91d42d4b330b097b51a5df5; no voters:
I20260620 20:14:59.209887 26333 leader_election.cc:288] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:59.209993 26343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:59.210705 26343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [term 1 LEADER]: Becoming Leader. State: Replica: 70da9b84e91d42d4b330b097b51a5df5, State: Running, Role: LEADER
I20260620 20:14:59.211284 26343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70da9b84e91d42d4b330b097b51a5df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } }
I20260620 20:14:59.212162 26333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:59.218765 26345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70da9b84e91d42d4b330b097b51a5df5. Latest consensus state: current_term: 1 leader_uuid: "70da9b84e91d42d4b330b097b51a5df5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70da9b84e91d42d4b330b097b51a5df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } } }
I20260620 20:14:59.219173 26345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:59.219756 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70da9b84e91d42d4b330b097b51a5df5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70da9b84e91d42d4b330b097b51a5df5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } } }
I20260620 20:14:59.220041 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:59.220198 26358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:59.225327 26358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:59.233285 26358 catalog_manager.cc:1347] Generated new cluster ID: ee9db7e4cbf945c09bc536d2f36a0a1d
I20260620 20:14:59.233397 26358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:59.252892 26358 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:59.254185 26358 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:59.261715 26358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70da9b84e91d42d4b330b097b51a5df5: Generated new TSK 0
I20260620 20:14:59.262555 26358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:59.275421 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26266
W20260620 20:14:59.276577 26328 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:54063
W20260620 20:14:59.277298 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26266: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:14:59.277436 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 26266
W20260620 20:14:59.293690 26251 connection.cc:535] client connection to 127.0.0.1:42499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42499): Connection reset by peer (error 104)
2026-06-20T20:14:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1930 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:59 dist-test-slave-4bwq krb5kdc[26364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:59 dist-test-slave-4bwq krb5kdc[26364](info): set up 2 sockets
Jun 20 20:14:59 dist-test-slave-4bwq krb5kdc[26364](info): commencing operation
krb5kdc: starting...
W20260620 20:15:00.411046 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:15:00 dist-test-slave-4bwq krb5kdc[26364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:15:00.674045 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:01.018225 26377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:01.018567 26377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:01.018736 26377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:01.026216 26377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:01.026345 26377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:01.026479 26377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:01.026556 26377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:01.026641 26377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:01.039533 26377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:01.040871 26377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:01.042224 26377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:01.054446 26382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:01.054350 26385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:01.054483 26383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:01.054737 26377 server_base.cc:1034] running on GCE node
I20260620 20:15:01.055876 26377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:01.057335 26377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:01.058534 26377 hybrid_clock.cc:648] HybridClock initialized: now 1781986501058481 us; error 41 us; skew 500 ppm
I20260620 20:15:01.058909 26377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:01 dist-test-slave-4bwq krb5kdc[26364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:01.068019 26377 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:15:01.069689 26377 webserver.cc:460] Webserver started at http://127.0.0.1:37847/ using document root <none> and password file <none>
I20260620 20:15:01.070493 26377 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:01.070645 26377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:01.070969 26377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:01.074185 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "865e5bcfd97749eb8b739fb8fdc58a58"
format_stamp: "Formatted at 2026-06-20 20:15:01 on dist-test-slave-4bwq"
server_key: "7fb690a5df3b3ba0aaecf871353d1b72"
server_key_iv: "ece5506829edb440fd1e0a581cf494e1"
server_key_version: "encryptionkey@0"
I20260620 20:15:01.075064 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "865e5bcfd97749eb8b739fb8fdc58a58"
format_stamp: "Formatted at 2026-06-20 20:15:01 on dist-test-slave-4bwq"
server_key: "7fb690a5df3b3ba0aaecf871353d1b72"
server_key_iv: "ece5506829edb440fd1e0a581cf494e1"
server_key_version: "encryptionkey@0"
I20260620 20:15:01.080593 26377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:15:01.084360 26392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:01.085850 26377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:15:01.086064 26377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "865e5bcfd97749eb8b739fb8fdc58a58"
format_stamp: "Formatted at 2026-06-20 20:15:01 on dist-test-slave-4bwq"
server_key: "7fb690a5df3b3ba0aaecf871353d1b72"
server_key_iv: "ece5506829edb440fd1e0a581cf494e1"
server_key_version: "encryptionkey@0"
I20260620 20:15:01.086294 26377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:01.092952 26377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:01.097663 26377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:01.097991 26377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:01.117604 26377 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40221
I20260620 20:15:01.117662 26443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40221 every 8 connection(s)
I20260620 20:15:01.119128 26377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:01.120098 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 26377
I20260620 20:15:01.120291 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:15:01.120831 4774 external_mini_cluster.cc:1428] Setting key 559cba8ff511118a80c6d25b1f173158
I20260620 20:15:01.123900 26444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:01.128947 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:01.122733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:54531 (local address 127.0.0.1:40221)
0620 20:15:01.123546 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:01.123558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:01.123624 (+ 66us) server_negotiation.cc:407] Connection header received
0620 20:15:01.125374 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:01.125397 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:01.125736 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:01.126144 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:01.127282 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:01.127288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:01.127301 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:01.127314 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:01.127394 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:01.127417 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:01.127671 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:01.127824 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 20:15:01.128011 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":347,"threads_started":1}
I20260620 20:15:01.132797 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:01.135546 26444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58: Bootstrap starting.
I20260620 20:15:01.139030 26444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:01.139828 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:01.137113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:46397 (local address 127.0.0.1:40221)
0620 20:15:01.137304 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:01.137309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:01.137843 (+ 534us) server_negotiation.cc:407] Connection header received
0620 20:15:01.137979 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:01.137985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:01.138057 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:01.138169 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:01.139162 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:01.139167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:01.139171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:01.139192 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:01.139243 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:01.139266 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:01.139403 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:01.139459 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 20:15:01.139534 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:15:01.140460 26444 log.cc:824] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:01.143268 26444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58: No bootstrap required, opened a new log
I20260620 20:15:01.150666 26444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865e5bcfd97749eb8b739fb8fdc58a58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40221 } }
I20260620 20:15:01.151098 26444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:01.151255 26444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865e5bcfd97749eb8b739fb8fdc58a58, State: Initialized, Role: FOLLOWER
I20260620 20:15:01.151827 26444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865e5bcfd97749eb8b739fb8fdc58a58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40221 } }
I20260620 20:15:01.152067 26444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:01.152220 26444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:01.152390 26444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:01.154634 26444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865e5bcfd97749eb8b739fb8fdc58a58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40221 } }
I20260620 20:15:01.155128 26444 leader_election.cc:302] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 865e5bcfd97749eb8b739fb8fdc58a58; no voters:
I20260620 20:15:01.155822 26444 leader_election.cc:288] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:01.155941 26454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:01.156713 26454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [term 1 LEADER]: Becoming Leader. State: Replica: 865e5bcfd97749eb8b739fb8fdc58a58, State: Running, Role: LEADER
I20260620 20:15:01.157310 26454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865e5bcfd97749eb8b739fb8fdc58a58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40221 } }
I20260620 20:15:01.158062 26444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:01.160066 26455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "865e5bcfd97749eb8b739fb8fdc58a58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865e5bcfd97749eb8b739fb8fdc58a58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40221 } } }
I20260620 20:15:01.160600 26455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:01.160089 26456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865e5bcfd97749eb8b739fb8fdc58a58. Latest consensus state: current_term: 1 leader_uuid: "865e5bcfd97749eb8b739fb8fdc58a58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865e5bcfd97749eb8b739fb8fdc58a58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40221 } } }
I20260620 20:15:01.163146 26456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:01.163760 26460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:01.169459 26460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:01.178655 26460 catalog_manager.cc:1347] Generated new cluster ID: cf3bd509e93c49fa81841fec010b11c8
I20260620 20:15:01.178872 26460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:01.193331 26460 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:01.193755 26460 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:01.202512 26460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 865e5bcfd97749eb8b739fb8fdc58a58: Generated new TSK 0
I20260620 20:15:01.203722 26460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:15:01.215787 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26377
W20260620 20:15:01.216888 26439 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:54531
W20260620 20:15:01.217638 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26377: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:01.217792 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 26377
2026-06-20T20:15:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:01 dist-test-slave-4bwq krb5kdc[26475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:01 dist-test-slave-4bwq krb5kdc[26475](info): set up 2 sockets
Jun 20 20:15:01 dist-test-slave-4bwq krb5kdc[26475](info): commencing operation
krb5kdc: starting...
W20260620 20:15:02.359202 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:15:02 dist-test-slave-4bwq krb5kdc[26475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:15:02.623751 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41913
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:40167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:41913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:02.964100 26488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:02.964471 26488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:02.964644 26488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:02.972136 26488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:02.972306 26488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:02.972424 26488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:02.972536 26488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:02.972615 26488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:02.983160 26488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:40167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:41913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:41913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:02.984542 26488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:02.985864 26488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:02.997784 26493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:02.997745 26494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:02.997953 26496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:02.999289 26488 server_base.cc:1034] running on GCE node
I20260620 20:15:03.000093 26488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:03.001616 26488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:03.002847 26488 hybrid_clock.cc:648] HybridClock initialized: now 1781986503002801 us; error 44 us; skew 500 ppm
I20260620 20:15:03.003228 26488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:03 dist-test-slave-4bwq krb5kdc[26475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986503, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:03.006625 26488 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:15:03.008078 26488 webserver.cc:460] Webserver started at http://127.4.169.190:40791/ using document root <none> and password file <none>
I20260620 20:15:03.008749 26488 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:03.008850 26488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:03.009099 26488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:03.012107 26488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "36f7226234ec4193b0fb5d901f05c919"
format_stamp: "Formatted at 2026-06-20 20:15:03 on dist-test-slave-4bwq"
server_key: "a1a31fdd43ef981b1da4cdff64e75e55"
server_key_iv: "59f58005188218c29f221fa8aeb69886"
server_key_version: "encryptionkey@0"
I20260620 20:15:03.012845 26488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "36f7226234ec4193b0fb5d901f05c919"
format_stamp: "Formatted at 2026-06-20 20:15:03 on dist-test-slave-4bwq"
server_key: "a1a31fdd43ef981b1da4cdff64e75e55"
server_key_iv: "59f58005188218c29f221fa8aeb69886"
server_key_version: "encryptionkey@0"
I20260620 20:15:03.018008 26488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:15:03.021484 26503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:03.022917 26488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:03.023101 26488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "36f7226234ec4193b0fb5d901f05c919"
format_stamp: "Formatted at 2026-06-20 20:15:03 on dist-test-slave-4bwq"
server_key: "a1a31fdd43ef981b1da4cdff64e75e55"
server_key_iv: "59f58005188218c29f221fa8aeb69886"
server_key_version: "encryptionkey@0"
I20260620 20:15:03.023304 26488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:03.056070 26488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:03.060225 26488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:03.060528 26488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:03.079756 26488 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:41913
I20260620 20:15:03.079764 26554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:41913 every 8 connection(s)
I20260620 20:15:03.081283 26488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:03.085211 26555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:03.090056 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 26488
I20260620 20:15:03.090276 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:15:03.090770 4774 external_mini_cluster.cc:1428] Setting key 8b8935f769c5b231378ee7d54ecd747f
I20260620 20:15:03.096843 26555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919: Bootstrap starting.
I20260620 20:15:03.100713 26555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:03.101939 26555 log.cc:824] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:03.105533 26555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919: No bootstrap required, opened a new log
Jun 20 20:15:03 dist-test-slave-4bwq krb5kdc[26475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:15:03.113840 26555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f7226234ec4193b0fb5d901f05c919" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41913 } }
I20260620 20:15:03.114364 26555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:03.114507 26555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f7226234ec4193b0fb5d901f05c919, State: Initialized, Role: FOLLOWER
I20260620 20:15:03.115136 26555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f7226234ec4193b0fb5d901f05c919" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41913 } }
I20260620 20:15:03.115398 26555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:03.115509 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:03.092691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38681 (local address 127.4.169.190:41913)
0620 20:15:03.093541 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:03.093554 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:03.093600 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:15:03.095489 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:03.095516 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:03.095966 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:03.096311 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:03.098191 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:03.099588 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:03.101445 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:03.102210 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:03.102341 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:03.107464 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:03.107472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:03.107487 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:03.107527 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:03.110786 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:03.111678 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:03.111684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:03.111691 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:03.111852 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:03.112354 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:03.112358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:03.112360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:03.112930 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:03.113261 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:03.113757 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 20:15:03.114083 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":408,"threads_started":1}
I20260620 20:15:03.115566 26555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:03.115754 26555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:03.118207 26555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f7226234ec4193b0fb5d901f05c919" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41913 } }
I20260620 20:15:03.118804 26555 leader_election.cc:302] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36f7226234ec4193b0fb5d901f05c919; no voters:
I20260620 20:15:03.119488 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:03.119469 26555 leader_election.cc:288] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:03.119592 26560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:03.122186 26555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:03.124029 26560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [term 1 LEADER]: Becoming Leader. State: Replica: 36f7226234ec4193b0fb5d901f05c919, State: Running, Role: LEADER
I20260620 20:15:03.124933 26560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f7226234ec4193b0fb5d901f05c919" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41913 } }
I20260620 20:15:03.128468 26562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36f7226234ec4193b0fb5d901f05c919. Latest consensus state: current_term: 1 leader_uuid: "36f7226234ec4193b0fb5d901f05c919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f7226234ec4193b0fb5d901f05c919" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41913 } } }
I20260620 20:15:03.128955 26562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:03.129598 26561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36f7226234ec4193b0fb5d901f05c919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f7226234ec4193b0fb5d901f05c919" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 41913 } } }
I20260620 20:15:03.129921 26561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:03.145697 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:03.124804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53549 (local address 127.4.169.190:41913)
0620 20:15:03.125091 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:03.125097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:03.125123 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:15:03.125177 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:03.125181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:03.125265 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:03.125364 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:03.127268 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:03.128422 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:03.130529 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:03.131507 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:03.131651 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:03.136064 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:03.136074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:03.136082 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:03.136142 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:03.140834 (+ 4692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:03.142107 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:03.142116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:03.142122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:03.142324 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:03.143057 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:03.143066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:03.143071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:03.143606 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:03.143924 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:03.144875 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 20:15:03.145147 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260620 20:15:03.147876 26570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:03.153920 26570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:03.163357 26570 catalog_manager.cc:1347] Generated new cluster ID: 1f4aeecab42248149b38319afbe99970
I20260620 20:15:03.163484 26570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:03.184214 26570 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:03.186120 26570 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:03.198287 26570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36f7226234ec4193b0fb5d901f05c919: Generated new TSK 0
I20260620 20:15:03.199465 26570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:15:03.211203 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26488
I20260620 20:15:03.212407 26550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:03.304324 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 26488
W20260620 20:15:03.321751 26473 connection.cc:535] client connection to 127.4.169.190:41913 recv error: Network error: failed to read from TLS socket (remote: 127.4.169.190:41913): Connection reset by peer (error 104)
2026-06-20T20:15:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:03 dist-test-slave-4bwq krb5kdc[26588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:03 dist-test-slave-4bwq krb5kdc[26588](info): set up 2 sockets
Jun 20 20:15:03 dist-test-slave-4bwq krb5kdc[26588](info): commencing operation
krb5kdc: starting...
W20260620 20:15:04.423173 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:15:04 dist-test-slave-4bwq krb5kdc[26588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:15:04.671342 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:37993
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:37993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:05.001678 26601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:05.002023 26601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:05.002202 26601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:05.010169 26601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:05.010344 26601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:05.010473 26601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:05.010578 26601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:05.010666 26601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:05.020625 26601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:37993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:37993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:05.021943 26601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:05.023340 26601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:05.036870 26606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:05.036919 26607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:05.036942 26609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:05.038548 26601 server_base.cc:1034] running on GCE node
I20260620 20:15:05.039470 26601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:05.040805 26601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:05.042032 26601 hybrid_clock.cc:648] HybridClock initialized: now 1781986505041985 us; error 41 us; skew 500 ppm
I20260620 20:15:05.042464 26601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:05 dist-test-slave-4bwq krb5kdc[26588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986505, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:05.045634 26601 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:15:05.047214 26601 webserver.cc:460] Webserver started at http://127.4.169.190:41309/ using document root <none> and password file <none>
I20260620 20:15:05.047875 26601 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:05.048005 26601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:05.048307 26601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:05.051287 26601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "1a71c7bbf18e4b7c91be0239f5df88ae"
format_stamp: "Formatted at 2026-06-20 20:15:05 on dist-test-slave-4bwq"
server_key: "19a30ab1d3c901738108eb9857e59b0a"
server_key_iv: "cc3495ae22de7491c39ff9ee94eaf9fd"
server_key_version: "encryptionkey@0"
I20260620 20:15:05.052038 26601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "1a71c7bbf18e4b7c91be0239f5df88ae"
format_stamp: "Formatted at 2026-06-20 20:15:05 on dist-test-slave-4bwq"
server_key: "19a30ab1d3c901738108eb9857e59b0a"
server_key_iv: "cc3495ae22de7491c39ff9ee94eaf9fd"
server_key_version: "encryptionkey@0"
I20260620 20:15:05.056995 26601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:15:05.060433 26616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:05.061697 26601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:15:05.061872 26601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "1a71c7bbf18e4b7c91be0239f5df88ae"
format_stamp: "Formatted at 2026-06-20 20:15:05 on dist-test-slave-4bwq"
server_key: "19a30ab1d3c901738108eb9857e59b0a"
server_key_iv: "cc3495ae22de7491c39ff9ee94eaf9fd"
server_key_version: "encryptionkey@0"
I20260620 20:15:05.062083 26601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:05.106320 26601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:05.110579 26601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:05.110879 26601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:05.130143 26601 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:37993
I20260620 20:15:05.130175 26667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:37993 every 8 connection(s)
I20260620 20:15:05.131531 26601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:05.135219 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:05.136175 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 26601
I20260620 20:15:05.136371 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:15:05.136929 4774 external_mini_cluster.cc:1428] Setting key 3389209bf9e32b59ab22c1b27dcfb120
I20260620 20:15:05.148862 26668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae: Bootstrap starting.
I20260620 20:15:05.150347 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:05.138777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43173 (local address 127.4.169.190:37993)
0620 20:15:05.139604 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:05.139616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:05.139659 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:15:05.141394 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:05.141418 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:05.141794 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:05.142138 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:05.142712 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:05.143952 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:05.145882 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:05.146718 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:05.146850 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:05.148124 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:05.148131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:05.148143 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:05.148658 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:05.148732 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:05.148755 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:05.148872 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:05.149214 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 20:15:05.149397 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":387,"threads_started":1}
I20260620 20:15:05.152412 26668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:05.153579 26668 log.cc:824] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:05.154201 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:05.156225 26668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae: No bootstrap required, opened a new log
I20260620 20:15:05.164642 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:05.158554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53677 (local address 127.4.169.190:37993)
0620 20:15:05.158737 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:05.158745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:05.159148 (+ 403us) server_negotiation.cc:407] Connection header received
0620 20:15:05.159353 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:05.159368 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:05.159425 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:05.159531 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:05.160023 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:05.160806 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:05.162089 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:05.162633 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:05.162794 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:05.163935 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:05.163939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:05.163943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:05.163958 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:05.164008 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:05.164022 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:05.164139 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:05.164333 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 20:15:05.164387 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:15:05.164336 26668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a71c7bbf18e4b7c91be0239f5df88ae" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37993 } }
I20260620 20:15:05.164819 26668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:05.164955 26668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a71c7bbf18e4b7c91be0239f5df88ae, State: Initialized, Role: FOLLOWER
I20260620 20:15:05.165619 26668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a71c7bbf18e4b7c91be0239f5df88ae" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37993 } }
I20260620 20:15:05.165896 26668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:05.166049 26668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:05.166232 26668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:05.168403 26668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a71c7bbf18e4b7c91be0239f5df88ae" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37993 } }
I20260620 20:15:05.168877 26668 leader_election.cc:302] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a71c7bbf18e4b7c91be0239f5df88ae; no voters:
I20260620 20:15:05.169401 26668 leader_election.cc:288] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:05.169530 26678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:05.170821 26678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [term 1 LEADER]: Becoming Leader. State: Replica: 1a71c7bbf18e4b7c91be0239f5df88ae, State: Running, Role: LEADER
I20260620 20:15:05.171350 26678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a71c7bbf18e4b7c91be0239f5df88ae" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37993 } }
I20260620 20:15:05.171568 26668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:05.174431 26680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a71c7bbf18e4b7c91be0239f5df88ae. Latest consensus state: current_term: 1 leader_uuid: "1a71c7bbf18e4b7c91be0239f5df88ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a71c7bbf18e4b7c91be0239f5df88ae" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37993 } } }
I20260620 20:15:05.174960 26680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:05.174222 26679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a71c7bbf18e4b7c91be0239f5df88ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a71c7bbf18e4b7c91be0239f5df88ae" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 37993 } } }
I20260620 20:15:05.175510 26679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:05.177335 26687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:05.182380 26687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:05.190815 26687 catalog_manager.cc:1347] Generated new cluster ID: 167df2146c974569bff19421ccb34ea3
I20260620 20:15:05.190927 26687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:05.216261 26687 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:05.217532 26687 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:05.228992 26687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a71c7bbf18e4b7c91be0239f5df88ae: Generated new TSK 0
I20260620 20:15:05.229789 26687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:15:05.242213 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26601
W20260620 20:15:05.243367 26663 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43173
W20260620 20:15:05.244177 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26601: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:05.244383 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 26601
W20260620 20:15:05.258340 26586 connection.cc:535] client connection to 127.4.169.190:37993 recv error: Network error: failed to read from TLS socket (remote: 127.4.169.190:37993): Connection reset by peer (error 104)
2026-06-20T20:15:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:05 dist-test-slave-4bwq krb5kdc[26699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:05 dist-test-slave-4bwq krb5kdc[26699](info): set up 2 sockets
Jun 20 20:15:05 dist-test-slave-4bwq krb5kdc[26699](info): commencing operation
krb5kdc: starting...
W20260620 20:15:06.363327 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:15:06 dist-test-slave-4bwq krb5kdc[26699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:15:06.632334 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:38593
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:38593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:06.991169 26712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:06.991489 26712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:06.991663 26712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:06.999433 26712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:06.999573 26712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:06.999697 26712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:06.999802 26712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:06.999878 26712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:07.010499 26712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:38593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:07.011929 26712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:07.013338 26712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:07.026046 26718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:07.026170 26717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:07.026072 26720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:07.026974 26712 server_base.cc:1034] running on GCE node
I20260620 20:15:07.027866 26712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:07.030041 26712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:07.031337 26712 hybrid_clock.cc:648] HybridClock initialized: now 1781986507031279 us; error 85 us; skew 500 ppm
I20260620 20:15:07.031906 26712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:07 dist-test-slave-4bwq krb5kdc[26699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:07.040382 26712 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:15:07.042115 26712 webserver.cc:460] Webserver started at http://127.4.169.190:46269/ using document root <none> and password file <none>
I20260620 20:15:07.042971 26712 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:07.043151 26712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:07.043501 26712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:07.046880 26712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "61180e44d2384cdca61b75a9aa6e2db2"
format_stamp: "Formatted at 2026-06-20 20:15:07 on dist-test-slave-4bwq"
server_key: "7bbc05a7b0ea003330950aae0016a919"
server_key_iv: "86d110d75b121592568e92c13bb70799"
server_key_version: "encryptionkey@0"
I20260620 20:15:07.047782 26712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "61180e44d2384cdca61b75a9aa6e2db2"
format_stamp: "Formatted at 2026-06-20 20:15:07 on dist-test-slave-4bwq"
server_key: "7bbc05a7b0ea003330950aae0016a919"
server_key_iv: "86d110d75b121592568e92c13bb70799"
server_key_version: "encryptionkey@0"
I20260620 20:15:07.053833 26712 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 20:15:07.058091 26727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:07.059791 26712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:07.060029 26712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "61180e44d2384cdca61b75a9aa6e2db2"
format_stamp: "Formatted at 2026-06-20 20:15:07 on dist-test-slave-4bwq"
server_key: "7bbc05a7b0ea003330950aae0016a919"
server_key_iv: "86d110d75b121592568e92c13bb70799"
server_key_version: "encryptionkey@0"
I20260620 20:15:07.060321 26712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:07.067390 26712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:07.072261 26712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:07.072625 26712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:07.093467 26712 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:38593
I20260620 20:15:07.093499 26778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:38593 every 8 connection(s)
I20260620 20:15:07.095132 26712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:07.099407 26779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:07.100205 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 26712
I20260620 20:15:07.100406 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:15:07.100986 4774 external_mini_cluster.cc:1428] Setting key 51962f8d9ac02a191abf20842a3c8333
I20260620 20:15:07.111227 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:07.103330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43689 (local address 127.4.169.190:38593)
0620 20:15:07.104539 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:07.104563 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:07.104947 (+ 384us) server_negotiation.cc:407] Connection header received
0620 20:15:07.106912 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:07.106948 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:07.107541 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:07.108068 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:07.109112 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:07.109121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:07.109138 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:07.109167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:07.109241 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:07.109273 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:07.109502 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:07.109627 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 20:15:07.109866 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":554,"threads_started":1}
I20260620 20:15:07.115123 26779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2: Bootstrap starting.
I20260620 20:15:07.115563 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:07.119400 26779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:07.120916 26779 log.cc:824] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:07.124078 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:07.121137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42331 (local address 127.4.169.190:38593)
0620 20:15:07.121325 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:07.121331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:07.121925 (+ 594us) server_negotiation.cc:407] Connection header received
0620 20:15:07.122077 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:07.122089 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:07.122158 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:07.122299 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:07.123350 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:07.123355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:07.123359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:07.123374 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:07.123415 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:07.123430 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:07.123527 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:07.123700 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 20:15:07.123755 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:15:07.124905 26779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2: No bootstrap required, opened a new log
I20260620 20:15:07.133000 26779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61180e44d2384cdca61b75a9aa6e2db2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38593 } }
I20260620 20:15:07.133453 26779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:07.133564 26779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61180e44d2384cdca61b75a9aa6e2db2, State: Initialized, Role: FOLLOWER
I20260620 20:15:07.134268 26779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61180e44d2384cdca61b75a9aa6e2db2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38593 } }
I20260620 20:15:07.134575 26779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:07.134742 26779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:07.134929 26779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:07.137651 26779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61180e44d2384cdca61b75a9aa6e2db2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38593 } }
I20260620 20:15:07.138228 26779 leader_election.cc:302] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61180e44d2384cdca61b75a9aa6e2db2; no voters:
I20260620 20:15:07.138998 26779 leader_election.cc:288] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:07.139151 26789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:07.140522 26789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [term 1 LEADER]: Becoming Leader. State: Replica: 61180e44d2384cdca61b75a9aa6e2db2, State: Running, Role: LEADER
I20260620 20:15:07.141223 26789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61180e44d2384cdca61b75a9aa6e2db2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38593 } }
I20260620 20:15:07.141495 26779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:07.144209 26790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61180e44d2384cdca61b75a9aa6e2db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61180e44d2384cdca61b75a9aa6e2db2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38593 } } }
I20260620 20:15:07.144728 26790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:07.144639 26791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61180e44d2384cdca61b75a9aa6e2db2. Latest consensus state: current_term: 1 leader_uuid: "61180e44d2384cdca61b75a9aa6e2db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61180e44d2384cdca61b75a9aa6e2db2" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38593 } } }
I20260620 20:15:07.144935 26791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:07.146610 26798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:07.152680 26798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:07.162774 26798 catalog_manager.cc:1347] Generated new cluster ID: 91068eab9959413cadb830b334c8bf74
I20260620 20:15:07.162976 26798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:07.180944 26798 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:07.181430 26798 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:07.192754 26798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61180e44d2384cdca61b75a9aa6e2db2: Generated new TSK 0
I20260620 20:15:07.193717 26798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:15:07.199524 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26712
W20260620 20:15:07.201504 26774 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43689
W20260620 20:15:07.202369 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26712: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:07.202584 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 26712
2026-06-20T20:15:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:07 dist-test-slave-4bwq krb5kdc[26810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:07 dist-test-slave-4bwq krb5kdc[26810](info): set up 2 sockets
Jun 20 20:15:07 dist-test-slave-4bwq krb5kdc[26810](info): commencing operation
krb5kdc: starting...
W20260620 20:15:08.311359 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:15:08 dist-test-slave-4bwq krb5kdc[26810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:15:08.575522 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45151
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:45151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:08.919503 26823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:08.919857 26823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:08.920035 26823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:08.927541 26823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:08.927685 26823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:08.927798 26823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:08.927907 26823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:08.927989 26823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:08.938417 26823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:45151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:45151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:08.939790 26823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:08.941107 26823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:08.953189 26831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:08.953200 26828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:08.953224 26829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:08.954075 26823 server_base.cc:1034] running on GCE node
I20260620 20:15:08.954982 26823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:08.956547 26823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:08.957794 26823 hybrid_clock.cc:648] HybridClock initialized: now 1781986508957756 us; error 38 us; skew 500 ppm
I20260620 20:15:08.958195 26823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:08 dist-test-slave-4bwq krb5kdc[26810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986508, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:08.961570 26823 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:15:08.963315 26823 webserver.cc:460] Webserver started at http://127.4.169.190:42501/ using document root <none> and password file <none>
I20260620 20:15:08.964027 26823 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:08.964205 26823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:08.964515 26823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:08.967764 26823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "a2986ed231ef427889eff767f34a526e"
format_stamp: "Formatted at 2026-06-20 20:15:08 on dist-test-slave-4bwq"
server_key: "00b1d15a31d0038ef8e4ec8071407a17"
server_key_iv: "03de50fdee73b7d66b055edf26b37e60"
server_key_version: "encryptionkey@0"
I20260620 20:15:08.968598 26823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "a2986ed231ef427889eff767f34a526e"
format_stamp: "Formatted at 2026-06-20 20:15:08 on dist-test-slave-4bwq"
server_key: "00b1d15a31d0038ef8e4ec8071407a17"
server_key_iv: "03de50fdee73b7d66b055edf26b37e60"
server_key_version: "encryptionkey@0"
I20260620 20:15:08.974095 26823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 20:15:08.977840 26838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:08.979353 26823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:08.979557 26823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "a2986ed231ef427889eff767f34a526e"
format_stamp: "Formatted at 2026-06-20 20:15:08 on dist-test-slave-4bwq"
server_key: "00b1d15a31d0038ef8e4ec8071407a17"
server_key_iv: "03de50fdee73b7d66b055edf26b37e60"
server_key_version: "encryptionkey@0"
I20260620 20:15:08.979779 26823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:08.997081 26823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:09.001833 26823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:09.002159 26823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:09.022475 26823 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:45151
I20260620 20:15:09.022497 26889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:45151 every 8 connection(s)
I20260620 20:15:09.024011 26823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:09.028335 26890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:09.032132 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 26823
I20260620 20:15:09.032327 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:15:09.032939 4774 external_mini_cluster.cc:1428] Setting key 2a9bfb701bfa29a4d2cec6aa5b6a503d
I20260620 20:15:09.045253 26890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e: Bootstrap starting.
I20260620 20:15:09.049239 26890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:09.050360 26890 log.cc:824] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:15:09 dist-test-slave-4bwq krb5kdc[26810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:15:09.053995 26890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e: No bootstrap required, opened a new log
I20260620 20:15:09.061633 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:09.036354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52417 (local address 127.4.169.190:45151)
0620 20:15:09.038346 (+ 1992us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:09.038364 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:09.038444 (+ 80us) server_negotiation.cc:407] Connection header received
0620 20:15:09.040617 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:09.040669 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:09.041338 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:09.041880 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:09.043445 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:09.045470 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:09.046931 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:09.048067 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:09.048272 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:09.053075 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:09.053085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:09.053102 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:09.053143 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:09.056835 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:09.057897 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:09.057905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:09.057913 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:09.058119 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:09.058571 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:09.058575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:09.058577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:09.059221 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:09.059472 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:09.059973 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 20:15:09.060452 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1666,"thread_start_us":1261,"threads_started":1}
I20260620 20:15:09.063567 26890 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2986ed231ef427889eff767f34a526e" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45151 } }
I20260620 20:15:09.063992 26890 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:09.064111 26890 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2986ed231ef427889eff767f34a526e, State: Initialized, Role: FOLLOWER
I20260620 20:15:09.064792 26890 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2986ed231ef427889eff767f34a526e" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45151 } }
I20260620 20:15:09.065102 26890 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:09.065286 26890 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:09.065487 26890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:09.066071 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:09.068943 26890 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2986ed231ef427889eff767f34a526e" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45151 } }
I20260620 20:15:09.069708 26890 leader_election.cc:302] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2986ed231ef427889eff767f34a526e; no voters:
I20260620 20:15:09.070497 26890 leader_election.cc:288] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:09.070685 26898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:09.072749 26898 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [term 1 LEADER]: Becoming Leader. State: Replica: a2986ed231ef427889eff767f34a526e, State: Running, Role: LEADER
I20260620 20:15:09.073330 26890 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:09.073596 26898 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2986ed231ef427889eff767f34a526e" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45151 } }
I20260620 20:15:09.080821 26900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2986ed231ef427889eff767f34a526e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2986ed231ef427889eff767f34a526e" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45151 } } }
I20260620 20:15:09.081305 26900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:09.081990 26901 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2986ed231ef427889eff767f34a526e. Latest consensus state: current_term: 1 leader_uuid: "a2986ed231ef427889eff767f34a526e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2986ed231ef427889eff767f34a526e" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 45151 } } }
I20260620 20:15:09.082417 26901 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:09.083042 26908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:09.089763 26908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:09.094969 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:09.072713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60465 (local address 127.4.169.190:45151)
0620 20:15:09.072930 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:09.072942 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:09.072969 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:15:09.073251 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:09.073256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:09.073354 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:09.073488 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:09.076134 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:09.077681 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:09.080300 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:09.081396 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:09.081582 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:09.085823 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:09.085830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:09.085835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:09.085882 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:09.090288 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:09.091471 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:09.091478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:09.091481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:09.091662 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:09.092178 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:09.092186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:09.092190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:09.092646 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:09.092935 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:09.094265 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0620 20:15:09.094575 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:15:09.100802 26908 catalog_manager.cc:1347] Generated new cluster ID: 2219d442305b4e87a530227630d09dd3
I20260620 20:15:09.100944 26908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:09.115070 26908 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:09.117327 26908 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:09.130420 26908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2986ed231ef427889eff767f34a526e: Generated new TSK 0
I20260620 20:15:09.131606 26908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:15:09.143314 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26823
I20260620 20:15:09.144798 26885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:09.237336 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 26823
2026-06-20T20:15:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2038 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:09 dist-test-slave-4bwq krb5kdc[26923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:09 dist-test-slave-4bwq krb5kdc[26923](info): set up 2 sockets
Jun 20 20:15:09 dist-test-slave-4bwq krb5kdc[26923](info): commencing operation
krb5kdc: starting...
W20260620 20:15:10.351089 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:15:10 dist-test-slave-4bwq krb5kdc[26923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:15:10.620848 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42347
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:42347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:10.964709 26936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:10.965047 26936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:10.965142 26936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:10.972591 26936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:10.972734 26936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:10.972821 26936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:10.972925 26936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:10.973013 26936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:10.983276 26936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:42347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:10.984675 26936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:10.986263 26936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:10.998665 26942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:10.998625 26941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:10.998721 26944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:10.999970 26936 server_base.cc:1034] running on GCE node
I20260620 20:15:11.000895 26936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:11.002416 26936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:11.003659 26936 hybrid_clock.cc:648] HybridClock initialized: now 1781986511003595 us; error 62 us; skew 500 ppm
I20260620 20:15:11.003988 26936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:11 dist-test-slave-4bwq krb5kdc[26923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:11.007206 26936 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:15:11.008664 26936 webserver.cc:460] Webserver started at http://127.4.169.190:41249/ using document root <none> and password file <none>
I20260620 20:15:11.009358 26936 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:11.009462 26936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:11.009732 26936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:11.012883 26936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "93349d596b7a4609a4de27f9c39dc207"
format_stamp: "Formatted at 2026-06-20 20:15:11 on dist-test-slave-4bwq"
server_key: "dcea99c6db2f9f06e089b879617085ea"
server_key_iv: "5b14834705db27e06365ab1f6ce4f398"
server_key_version: "encryptionkey@0"
I20260620 20:15:11.013659 26936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "93349d596b7a4609a4de27f9c39dc207"
format_stamp: "Formatted at 2026-06-20 20:15:11 on dist-test-slave-4bwq"
server_key: "dcea99c6db2f9f06e089b879617085ea"
server_key_iv: "5b14834705db27e06365ab1f6ce4f398"
server_key_version: "encryptionkey@0"
I20260620 20:15:11.019137 26936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:15:11.022704 26951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:11.024148 26936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:15:11.024286 26936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "93349d596b7a4609a4de27f9c39dc207"
format_stamp: "Formatted at 2026-06-20 20:15:11 on dist-test-slave-4bwq"
server_key: "dcea99c6db2f9f06e089b879617085ea"
server_key_iv: "5b14834705db27e06365ab1f6ce4f398"
server_key_version: "encryptionkey@0"
I20260620 20:15:11.024453 26936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:11.073470 26936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:11.078459 26936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:11.078724 26936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:11.098775 26936 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:42347
I20260620 20:15:11.098795 27002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:42347 every 8 connection(s)
I20260620 20:15:11.100239 26936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:11.100663 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 26936
I20260620 20:15:11.100832 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:15:11.101444 4774 external_mini_cluster.cc:1428] Setting key f6c0b3ecf105b52ccaa392534b5aafc0
I20260620 20:15:11.104780 27003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:11.115620 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:11.103504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40433 (local address 127.4.169.190:42347)
0620 20:15:11.104309 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:11.104321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:11.104361 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:15:11.106141 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:11.106163 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:11.106629 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:11.106962 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:11.107626 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:11.108996 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:11.110921 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:11.111667 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:11.111808 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:11.113105 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:11.113112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:11.113127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:11.113699 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:11.113774 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:11.113803 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:11.113898 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:11.114169 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 20:15:11.114507 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":363,"threads_started":1}
I20260620 20:15:11.117910 27003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207: Bootstrap starting.
I20260620 20:15:11.119747 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:11.121409 27003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:11.122524 27003 log.cc:824] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:11.125761 27003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207: No bootstrap required, opened a new log
I20260620 20:15:11.131235 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:11.124380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59587 (local address 127.4.169.190:42347)
0620 20:15:11.124553 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:11.124559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:11.124961 (+ 402us) server_negotiation.cc:407] Connection header received
0620 20:15:11.125124 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:11.125129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:11.125214 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:11.125328 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:11.125796 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:11.126916 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:11.128329 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:11.129195 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:11.129323 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:11.130456 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:11.130462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:11.130467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:11.130490 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:11.130535 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:11.130555 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:11.130638 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:11.130773 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:15:11.130866 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:15:11.135114 27003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93349d596b7a4609a4de27f9c39dc207" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42347 } }
I20260620 20:15:11.135505 27003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:11.135586 27003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93349d596b7a4609a4de27f9c39dc207, State: Initialized, Role: FOLLOWER
I20260620 20:15:11.136102 27003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93349d596b7a4609a4de27f9c39dc207" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42347 } }
I20260620 20:15:11.136358 27003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:11.136530 27003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:11.136724 27003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:11.139093 27003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93349d596b7a4609a4de27f9c39dc207" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42347 } }
I20260620 20:15:11.139571 27003 leader_election.cc:302] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93349d596b7a4609a4de27f9c39dc207; no voters:
I20260620 20:15:11.140283 27003 leader_election.cc:288] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:11.140494 27013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:11.141686 27013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [term 1 LEADER]: Becoming Leader. State: Replica: 93349d596b7a4609a4de27f9c39dc207, State: Running, Role: LEADER
I20260620 20:15:11.142549 27013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93349d596b7a4609a4de27f9c39dc207" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42347 } }
I20260620 20:15:11.144070 27003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:11.151091 27014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93349d596b7a4609a4de27f9c39dc207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93349d596b7a4609a4de27f9c39dc207" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42347 } } }
I20260620 20:15:11.151147 27015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93349d596b7a4609a4de27f9c39dc207. Latest consensus state: current_term: 1 leader_uuid: "93349d596b7a4609a4de27f9c39dc207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93349d596b7a4609a4de27f9c39dc207" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42347 } } }
I20260620 20:15:11.152866 27015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:11.152866 27014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:11.154549 27022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:11.166646 27022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:11.180989 27022 catalog_manager.cc:1347] Generated new cluster ID: 423325862c264d9a8430d71c6b688b26
I20260620 20:15:11.181139 27022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:11.204766 27022 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:11.206247 27022 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:11.213790 27022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93349d596b7a4609a4de27f9c39dc207: Generated new TSK 0
I20260620 20:15:11.214717 27022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:15:11.233078 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26936
W20260620 20:15:11.234195 26998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40433
W20260620 20:15:11.234912 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 26936: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:11.235054 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 26936
2026-06-20T20:15:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:11 dist-test-slave-4bwq krb5kdc[27034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:11 dist-test-slave-4bwq krb5kdc[27034](info): set up 2 sockets
Jun 20 20:15:11 dist-test-slave-4bwq krb5kdc[27034](info): commencing operation
krb5kdc: starting...
W20260620 20:15:12.351104 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:15:12 dist-test-slave-4bwq krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:15:12.617992 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:38595
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:12.955150 27047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:12.955456 27047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:12.955646 27047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:12.963117 27047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:12.963276 27047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:12.963387 27047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:12.963500 27047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:12.963581 27047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:12.973809 27047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:12.975188 27047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:12.976539 27047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:12.988933 27052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:12.989019 27053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:12.988957 27055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:12.990046 27047 server_base.cc:1034] running on GCE node
I20260620 20:15:12.990983 27047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:12.992445 27047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:12.993661 27047 hybrid_clock.cc:648] HybridClock initialized: now 1781986512993608 us; error 40 us; skew 500 ppm
I20260620 20:15:12.994041 27047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:13 dist-test-slave-4bwq krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986513, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:13.004324 27047 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:15:13.006033 27047 webserver.cc:460] Webserver started at http://127.4.169.190:32883/ using document root <none> and password file <none>
I20260620 20:15:13.006843 27047 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:13.006979 27047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:13.007314 27047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:13.010372 27047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "f43039e55a9e46599a980f100ae7a2b9"
format_stamp: "Formatted at 2026-06-20 20:15:13 on dist-test-slave-4bwq"
server_key: "39d7b6830f4f6422205418ac59765a6b"
server_key_iv: "ff544d0d6958ce6ca6b0e89297ede6d4"
server_key_version: "encryptionkey@0"
I20260620 20:15:13.011258 27047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "f43039e55a9e46599a980f100ae7a2b9"
format_stamp: "Formatted at 2026-06-20 20:15:13 on dist-test-slave-4bwq"
server_key: "39d7b6830f4f6422205418ac59765a6b"
server_key_iv: "ff544d0d6958ce6ca6b0e89297ede6d4"
server_key_version: "encryptionkey@0"
I20260620 20:15:13.016611 27047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:15:13.020256 27062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:13.021587 27047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:15:13.021766 27047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "f43039e55a9e46599a980f100ae7a2b9"
format_stamp: "Formatted at 2026-06-20 20:15:13 on dist-test-slave-4bwq"
server_key: "39d7b6830f4f6422205418ac59765a6b"
server_key_iv: "ff544d0d6958ce6ca6b0e89297ede6d4"
server_key_version: "encryptionkey@0"
I20260620 20:15:13.021973 27047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:13.027786 27047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:13.032227 27047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:13.032523 27047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:13.052603 27047 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:38595
I20260620 20:15:13.052644 27113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:38595 every 8 connection(s)
I20260620 20:15:13.054111 27047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:13.057897 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:13.063506 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 27047
I20260620 20:15:13.063709 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:15:13.064136 4774 external_mini_cluster.cc:1428] Setting key 13fd9ca925654e080a7e3286735c7041
I20260620 20:15:13.069023 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9: Bootstrap starting.
I20260620 20:15:13.072124 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:13.065975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48707 (local address 127.4.169.190:38595)
0620 20:15:13.066912 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:13.066925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:13.066969 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:15:13.068655 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:13.068679 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:13.069024 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:13.069409 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:13.070527 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:13.070534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:13.070547 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:13.070567 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:13.070630 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:13.070656 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:13.070891 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:13.070995 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 20:15:13.071249 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":453,"threads_started":1}
I20260620 20:15:13.072705 27114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:13.073805 27114 log.cc:824] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:13.075798 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:13.076817 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9: No bootstrap required, opened a new log
I20260620 20:15:13.082849 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:13.080324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41129 (local address 127.4.169.190:38595)
0620 20:15:13.080541 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:13.080548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:13.080867 (+ 319us) server_negotiation.cc:407] Connection header received
0620 20:15:13.081039 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:13.081043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:13.081093 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:13.081177 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:13.082137 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:13.082141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:13.082144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:15:13.082159 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:13.082221 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:15:13.082237 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:13.082338 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:13.082487 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 20:15:13.082536 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:15:13.085882 27114 raft_consensus.cc:348] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43039e55a9e46599a980f100ae7a2b9" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38595 } }
I20260620 20:15:13.086294 27114 raft_consensus.cc:374] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:13.086424 27114 raft_consensus.cc:729] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43039e55a9e46599a980f100ae7a2b9, State: Initialized, Role: FOLLOWER
I20260620 20:15:13.087028 27114 consensus_queue.cc:260] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43039e55a9e46599a980f100ae7a2b9" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38595 } }
I20260620 20:15:13.087293 27114 raft_consensus.cc:388] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:13.087451 27114 raft_consensus.cc:482] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:13.087610 27114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:13.089874 27114 raft_consensus.cc:504] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43039e55a9e46599a980f100ae7a2b9" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38595 } }
I20260620 20:15:13.090512 27114 leader_election.cc:302] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f43039e55a9e46599a980f100ae7a2b9; no voters:
I20260620 20:15:13.091167 27114 leader_election.cc:288] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:13.091506 27124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:13.092573 27124 raft_consensus.cc:686] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [term 1 LEADER]: Becoming Leader. State: Replica: f43039e55a9e46599a980f100ae7a2b9, State: Running, Role: LEADER
I20260620 20:15:13.093221 27124 consensus_queue.cc:237] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43039e55a9e46599a980f100ae7a2b9" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38595 } }
I20260620 20:15:13.093514 27114 sys_catalog.cc:564] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:13.096117 27126 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f43039e55a9e46599a980f100ae7a2b9. Latest consensus state: current_term: 1 leader_uuid: "f43039e55a9e46599a980f100ae7a2b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43039e55a9e46599a980f100ae7a2b9" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38595 } } }
I20260620 20:15:13.096503 27126 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:13.096237 27125 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f43039e55a9e46599a980f100ae7a2b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43039e55a9e46599a980f100ae7a2b9" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 38595 } } }
I20260620 20:15:13.096681 27125 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:13.097463 27132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:13.109351 27132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:13.117790 27132 catalog_manager.cc:1347] Generated new cluster ID: 9756f093fced423db36b5cb75538bd0b
I20260620 20:15:13.117923 27132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:13.150601 27132 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:13.151048 27132 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:13.162375 27132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f43039e55a9e46599a980f100ae7a2b9: Generated new TSK 0
I20260620 20:15:13.163328 27132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:15:13.176390 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 27047
W20260620 20:15:13.178288 27109 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48707
W20260620 20:15:13.179018 4774 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 27047: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:13.179196 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 27047
2026-06-20T20:15:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1952 ms)
[----------] 18 tests from AuthTokenIssuingTest (35495 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:13 dist-test-slave-4bwq krb5kdc[27145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:13 dist-test-slave-4bwq krb5kdc[27145](info): set up 2 sockets
Jun 20 20:15:13 dist-test-slave-4bwq krb5kdc[27145](info): commencing operation
krb5kdc: starting...
W20260620 20:15:14.295245 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:15:14 dist-test-slave-4bwq krb5kdc[27145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:15:14.557760 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34807
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34807,127.4.169.189:41803,127.4.169.188:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:14.893038 27158 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:15:14.893458 27158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:14.893586 27158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:14.893733 27158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:14.901401 27158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:14.901531 27158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:14.901669 27158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:14.901789 27158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:14.901907 27158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:14.912395 27158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34807,127.4.169.189:41803,127.4.169.188:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:34807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:14.913813 27158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:14.915197 27158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:14.927461 27163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:14.927423 27166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:14.927387 27164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:14.928773 27158 server_base.cc:1034] running on GCE node
I20260620 20:15:14.929653 27158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:14.931449 27158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:14.932744 27158 hybrid_clock.cc:648] HybridClock initialized: now 1781986514932652 us; error 58 us; skew 500 ppm
I20260620 20:15:14.933113 27158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:14 dist-test-slave-4bwq krb5kdc[27145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:14.936437 27158 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:15:14.937920 27158 webserver.cc:460] Webserver started at http://127.4.169.190:45825/ using document root <none> and password file <none>
I20260620 20:15:14.938666 27158 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:14.938755 27158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:14.939014 27158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:14.941946 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "5038613807a545b59916b7f7c3d37567"
format_stamp: "Formatted at 2026-06-20 20:15:14 on dist-test-slave-4bwq"
server_key: "d77c3f3eb92eca7addc46b74b50accf2"
server_key_iv: "7df545e9d9da5c3cb3a3946c7a094558"
server_key_version: "encryptionkey@0"
I20260620 20:15:14.942739 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "5038613807a545b59916b7f7c3d37567"
format_stamp: "Formatted at 2026-06-20 20:15:14 on dist-test-slave-4bwq"
server_key: "d77c3f3eb92eca7addc46b74b50accf2"
server_key_iv: "7df545e9d9da5c3cb3a3946c7a094558"
server_key_version: "encryptionkey@0"
I20260620 20:15:14.947683 27158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:15:14.950970 27173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:14.952342 27158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:15:14.952519 27158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "5038613807a545b59916b7f7c3d37567"
format_stamp: "Formatted at 2026-06-20 20:15:14 on dist-test-slave-4bwq"
server_key: "d77c3f3eb92eca7addc46b74b50accf2"
server_key_iv: "7df545e9d9da5c3cb3a3946c7a094558"
server_key_version: "encryptionkey@0"
I20260620 20:15:14.952724 27158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:14.966322 27158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:14.970623 27158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:14.970912 27158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:14.990232 27158 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:34807
I20260620 20:15:14.990252 27224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:34807 every 8 connection(s)
I20260620 20:15:14.991694 27158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:14.992934 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 27158
I20260620 20:15:14.993152 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:15:14.993803 4774 external_mini_cluster.cc:1428] Setting key fd5615149304e050f7ee415e9f20e6d8
I20260620 20:15:14.995842 27225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:15.002800 27225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:15 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.169.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.189@KRBTEST.COM" created.
I20260620 20:15:15.026954 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.006089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.4.169.190:34807)
0620 20:15:15.007970 (+ 1881us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:15.007980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:15.008040 (+ 60us) server_negotiation.cc:407] Connection header received
0620 20:15:15.009683 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:15.009694 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:15.010005 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:15.010219 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:15.011503 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.013431 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.015000 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.015608 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.015818 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:15.019524 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:15.019532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:15.019545 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:15.019575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:15.022575 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:15.023281 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:15.023288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:15.023290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:15.023435 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:15.023831 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:15.023836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:15.023845 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:15.024336 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:15.024540 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:15.025037 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 20:15:15.025401 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1670,"spinlock_wait_cycles":46976,"thread_start_us":1451,"threads_started":1}
I20260620 20:15:15.026988 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.006090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34807 (local address 127.0.0.1:57926)
0620 20:15:15.007108 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.007169 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:15.009277 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:15.010251 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:15.010256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:15.010923 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:15.011272 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:15.011290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.013601 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.013605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:15.014853 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:15.014865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.015849 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.015855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:15.016026 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:15.016729 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:15.016752 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:15.019282 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:15.022724 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:15.022735 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:15.022748 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:15.023129 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:15.023554 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:15.023558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:15.023562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:15.023720 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:15.024477 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:15.024486 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:15.024685 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 20:15:15.024941 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 20:15:15.025506 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":136,"thread_start_us":377,"threads_started":1}
I20260620 20:15:15.029855 27225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:15.031038 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.030531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43176)
0620 20:15:15.030725 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.030867 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 20:15:15.031306 27176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
I20260620 20:15:15.034366 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.033484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43186)
0620 20:15:15.034077 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.034183 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 20:15:15.034797 27225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:41803: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab.
Entry for principal kudu/127.4.169.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab.
I20260620 20:15:15.054829 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.189:41803
--webserver_interface=127.4.169.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab
--principal=kudu/127.4.169.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34807,127.4.169.189:41803,127.4.169.188:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:15:15.084291 27225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } attempt: 1
I20260620 20:15:15.085511 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.085045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43190)
0620 20:15:15.085256 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.085378 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 20:15:15.086607 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.086198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43206)
0620 20:15:15.086360 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.086480 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 20:15:15.086788 27225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:41803: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
I20260620 20:15:15.155035 27225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } attempt: 2
I20260620 20:15:15.156323 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.155732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43208)
0620 20:15:15.156020 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.156162 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 20:15:15.157406 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.156982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43210)
0620 20:15:15.157147 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.157258 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:15:15.157569 27225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:41803: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
I20260620 20:15:15.248843 27225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } attempt: 3
I20260620 20:15:15.250056 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.249590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43216)
0620 20:15:15.249810 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.249908 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 20:15:15.251178 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.250814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43232)
0620 20:15:15.250964 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.251038 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:15:15.251363 27225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:41803: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
I20260620 20:15:15.394583 27225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } attempt: 4
I20260620 20:15:15.395901 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.395344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43248)
0620 20:15:15.395657 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.395763 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 20:15:15.396939 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.396568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43262)
0620 20:15:15.396739 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.396809 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:15:15.397166 27225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:41803: Network error: Client connection negotiation failed: client connection to 127.4.169.189:41803: connect: Connection refused (error 111)
W20260620 20:15:15.400283 27232 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:15:15.400672 27232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:15.400797 27232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:15.400933 27232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:15.408798 27232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:15.408927 27232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:15.409052 27232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:15.409142 27232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:15.409253 27232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:15.420055 27232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34807,127.4.169.189:41803,127.4.169.188:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab
--principal=kudu/127.4.169.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.189:41803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.169.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:15.421478 27232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:15.422992 27232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:15.435678 27240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:15.435760 27237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:15.435755 27238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:15.436518 27232 server_base.cc:1034] running on GCE node
I20260620 20:15:15.437316 27232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:15.438864 27232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:15.440109 27232 hybrid_clock.cc:648] HybridClock initialized: now 1781986515440036 us; error 51 us; skew 500 ppm
I20260620 20:15:15.440438 27232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:15 dist-test-slave-4bwq krb5kdc[27145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:15.443966 27232 init.cc:377] Logged in from keytab as kudu/127.4.169.189@KRBTEST.COM (short username kudu)
I20260620 20:15:15.445470 27232 webserver.cc:460] Webserver started at http://127.4.169.189:40385/ using document root <none> and password file <none>
I20260620 20:15:15.446158 27232 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:15.446290 27232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:15.446668 27232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:15.449635 27232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/data/instance:
uuid: "ac4f9f471a874097bb6a8365e97d8643"
format_stamp: "Formatted at 2026-06-20 20:15:15 on dist-test-slave-4bwq"
server_key: "0585ce6a5f85d87f141b89a68f0b25f4"
server_key_iv: "699fa76e88033bc71688e465d921952f"
server_key_version: "encryptionkey@0"
I20260620 20:15:15.450615 27232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/wal/instance:
uuid: "ac4f9f471a874097bb6a8365e97d8643"
format_stamp: "Formatted at 2026-06-20 20:15:15 on dist-test-slave-4bwq"
server_key: "0585ce6a5f85d87f141b89a68f0b25f4"
server_key_iv: "699fa76e88033bc71688e465d921952f"
server_key_version: "encryptionkey@0"
I20260620 20:15:15.455777 27232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:15:15.459570 27247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:15.461232 27232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:15:15.461443 27232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/wal
uuid: "ac4f9f471a874097bb6a8365e97d8643"
format_stamp: "Formatted at 2026-06-20 20:15:15 on dist-test-slave-4bwq"
server_key: "0585ce6a5f85d87f141b89a68f0b25f4"
server_key_iv: "699fa76e88033bc71688e465d921952f"
server_key_version: "encryptionkey@0"
I20260620 20:15:15.461694 27232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:15.483664 27232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:15.488368 27232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:15.488672 27232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:15.508180 27232 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.189:41803
I20260620 20:15:15.508200 27298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.189:41803 every 8 connection(s)
I20260620 20:15:15.509646 27232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
I20260620 20:15:15.511158 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 27232
I20260620 20:15:15.511373 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-1/wal/instance
I20260620 20:15:15.511896 4774 external_mini_cluster.cc:1428] Setting key 2fafe44075aff2553e31a38ca5210fde
I20260620 20:15:15.514439 27299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:15.524482 27299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:15 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.169.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.188@KRBTEST.COM" created.
I20260620 20:15:15.544524 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.527408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.4.169.190:34807)
0620 20:15:15.527585 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:15.527590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:15.528895 (+ 1305us) server_negotiation.cc:407] Connection header received
0620 20:15:15.530871 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:15.530877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:15.530969 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:15.531078 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:15.532567 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.533258 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.534813 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.535227 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.535366 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:15.538800 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:15.538809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:15.538818 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:15.538850 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:15.541602 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:15.542286 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:15.542292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:15.542296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:15.542473 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:15.542881 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:15.542886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:15.542889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:15.543161 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:15.543315 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:15.543987 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 20:15:15.544203 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:15:15.545465 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.527994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34807 (local address 127.0.0.1:57938)
0620 20:15:15.528729 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.528775 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:15.530599 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:15.531322 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:15.531331 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:15.531836 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:15.532397 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:15.532414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.533421 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.533425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:15.534664 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:15.534676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.535450 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.535458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:15.535693 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:15.536412 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:15.536433 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:15.538576 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:15.541742 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:15.541750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:15.541766 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:15.542158 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:15.542584 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:15.542589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:15.542593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:15.542768 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:15.543287 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:15.543294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:15.543632 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 20:15:15.543885 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 20:15:15.544377 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":299,"threads_started":1}
I20260620 20:15:15.547272 27299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:15 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:15:15.565063 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.547913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43270)
0620 20:15:15.548812 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.548838 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:15.548961 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:15.549386 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:15.549398 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:15.549907 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:15.550059 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:15.550067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.552233 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.552237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:15.553083 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:15.553094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.554308 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.554313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:15.554569 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:15.555200 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:15.555218 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:15.557665 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:15.562195 (+ 4530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:15.562199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:15.562202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:15.562668 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:15.563178 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:15.563192 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:15.563194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:15.563347 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:15.564236 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:15.564240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:15.564362 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 20:15:15.564481 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 20:15:15.564709 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":40}
I20260620 20:15:15.565511 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.548139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.4.169.189:41803)
0620 20:15:15.548845 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:15.548851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:15.548908 (+ 57us) server_negotiation.cc:407] Connection header received
0620 20:15:15.549062 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:15.549067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:15.549237 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:15.549344 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:15.550224 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.552071 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.553225 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.554143 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.554285 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:15.557873 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:15.557889 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:15.557911 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:15.557956 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:15.562044 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:15.562810 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:15.562823 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:15.562829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:15.562998 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:15.563476 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:15.563507 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:15.563512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:15.564086 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:15.564331 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:15.564695 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 20:15:15.565000 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":545,"thread_start_us":401,"threads_started":1}
I20260620 20:15:15.568058 27299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:15.569231 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.568754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54210)
0620 20:15:15.568945 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.569080 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 20:15:15.569474 27251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
I20260620 20:15:15.571219 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.570750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54220)
0620 20:15:15.570904 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.571001 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:15:15.571483 27299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:39307: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab.
Entry for principal kudu/127.4.169.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab.
I20260620 20:15:15.575595 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.188:39307
--webserver_interface=127.4.169.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab
--principal=kudu/127.4.169.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:34807,127.4.169.189:41803,127.4.169.188:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:15:15.620898 27299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } attempt: 1
I20260620 20:15:15.622299 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.621684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54236)
0620 20:15:15.621919 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.622097 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 20:15:15.623488 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.623027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54244)
0620 20:15:15.623232 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.623336 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 20:15:15.623754 27299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:39307: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
I20260620 20:15:15.692044 27299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } attempt: 2
I20260620 20:15:15.693354 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.692787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54250)
0620 20:15:15.693024 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.693148 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 20:15:15.694646 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.694151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54260)
0620 20:15:15.694333 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.694479 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 20:15:15.694886 27299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:39307: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
I20260620 20:15:15.696403 27225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } attempt: 5
Jun 20 20:15:15 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:15:15.711056 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.697193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.4.169.189:41803)
0620 20:15:15.697472 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:15.697480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:15.697502 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:15:15.697893 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:15.697898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:15.697963 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:15.698077 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:15.699004 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.699715 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.700939 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.701388 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.701530 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:15.705251 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:15.705256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:15.705260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:15.705297 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:15.708150 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:15.708949 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:15.708956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:15.708960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:15.709094 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:15.709510 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:15.709517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:15.709520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:15.709831 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:15.709995 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:15.710481 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 20:15:15.710722 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260620 20:15:15.711055 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.697192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43278)
0620 20:15:15.697420 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.697441 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:15.697678 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:15.698180 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:15.698185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:15.698653 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:15.698847 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:15.698856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.699877 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.699887 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:15.700775 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:15.700787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:15.701548 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:15.701555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:15.701798 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:15.702552 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:15.702573 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:15.705020 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:15.708315 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:15.708323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:15.708327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:15.708808 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:15.709230 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:15.709236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:15.709239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:15.709394 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:15.709957 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:15.709962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:15.710140 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 20:15:15.710328 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 20:15:15.710680 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 20:15:15.712399 27225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:15.713603 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.713132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54270)
0620 20:15:15.713357 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.713449 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 20:15:15.714721 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.714327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54280)
0620 20:15:15.714516 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.714607 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 20:15:15.715021 27225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:39307: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
I20260620 20:15:15.766281 27225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } attempt: 1
I20260620 20:15:15.767735 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.767215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54284)
0620 20:15:15.767446 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.767553 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 20:15:15.768811 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.768446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54290)
0620 20:15:15.768602 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.768686 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:15:15.769073 27225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:39307: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
I20260620 20:15:15.786130 27299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } attempt: 3
I20260620 20:15:15.787635 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.786951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54292)
0620 20:15:15.787226 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.787365 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 20:15:15.789211 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.788544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54306)
0620 20:15:15.788854 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.788990 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260620 20:15:15.789600 27299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:39307: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
I20260620 20:15:15.837311 27225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } attempt: 2
I20260620 20:15:15.838672 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.838017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54318)
0620 20:15:15.838291 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.838451 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 20:15:15.839869 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.839422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54334)
0620 20:15:15.839593 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.839711 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:15:15.840109 27225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:39307: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
I20260620 20:15:15.932874 27299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } attempt: 4
I20260620 20:15:15.934149 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.933557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54348)
0620 20:15:15.933837 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.933949 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 20:15:15.935333 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.934959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54354)
0620 20:15:15.935113 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.935201 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:15:15.935581 27299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:39307: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
W20260620 20:15:15.937114 27306 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:15:15.937477 27306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:15.937583 27306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:15.937716 27306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:15.945231 27306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:15.945312 27306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:15.945416 27306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:15.945494 27306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:15.945597 27306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:15.946318 27225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } attempt: 3
I20260620 20:15:15.947430 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.946928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54364)
0620 20:15:15.947132 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.947235 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 20:15:15.948555 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:15.948128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54378)
0620 20:15:15.948338 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:15.948421 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 20:15:15.948767 27225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:39307: Network error: Client connection negotiation failed: client connection to 127.4.169.188:39307: connect: Connection refused (error 111)
I20260620 20:15:15.956669 27306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:34807,127.4.169.189:41803,127.4.169.188:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab
--principal=kudu/127.4.169.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.188:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.169.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:15.958035 27306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:15.959498 27306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:15.972556 27311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:15.972556 27312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:15.972674 27314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:15.973474 27306 server_base.cc:1034] running on GCE node
I20260620 20:15:15.974463 27306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:15.976051 27306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:15.977332 27306 hybrid_clock.cc:648] HybridClock initialized: now 1781986515977249 us; error 77 us; skew 500 ppm
I20260620 20:15:15.977705 27306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:15 dist-test-slave-4bwq krb5kdc[27145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:15.981339 27306 init.cc:377] Logged in from keytab as kudu/127.4.169.188@KRBTEST.COM (short username kudu)
I20260620 20:15:15.983065 27306 webserver.cc:460] Webserver started at http://127.4.169.188:38805/ using document root <none> and password file <none>
I20260620 20:15:15.983789 27306 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:15.983892 27306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:15.984172 27306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:15.987366 27306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/data/instance:
uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b"
format_stamp: "Formatted at 2026-06-20 20:15:15 on dist-test-slave-4bwq"
server_key: "aac22571755b2c8fb4b96b35344ddcb7"
server_key_iv: "05c9764c7bf077ce25857e042b8436ab"
server_key_version: "encryptionkey@0"
I20260620 20:15:15.988157 27306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/wal/instance:
uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b"
format_stamp: "Formatted at 2026-06-20 20:15:15 on dist-test-slave-4bwq"
server_key: "aac22571755b2c8fb4b96b35344ddcb7"
server_key_iv: "05c9764c7bf077ce25857e042b8436ab"
server_key_version: "encryptionkey@0"
I20260620 20:15:15.995045 27306 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 20:15:15.999913 27321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:16.001534 27306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260620 20:15:16.001753 27306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/wal
uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b"
format_stamp: "Formatted at 2026-06-20 20:15:15 on dist-test-slave-4bwq"
server_key: "aac22571755b2c8fb4b96b35344ddcb7"
server_key_iv: "05c9764c7bf077ce25857e042b8436ab"
server_key_version: "encryptionkey@0"
I20260620 20:15:16.001979 27306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:16.034575 27306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:16.039377 27306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:16.039685 27306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:16.059264 27306 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.188:39307
I20260620 20:15:16.059330 27372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.188:39307 every 8 connection(s)
I20260620 20:15:16.060746 27306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
I20260620 20:15:16.062667 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 27306
I20260620 20:15:16.062868 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986152907614-4774-0/minicluster-data/master-2/wal/instance
I20260620 20:15:16.063295 4774 external_mini_cluster.cc:1428] Setting key 80e80f5b5f7106a59e93411f1e67f69d
I20260620 20:15:16.065513 27373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:16.075847 27373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:16 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
I20260620 20:15:16.097129 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.065325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.4.169.188:39307)
0620 20:15:16.066217 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.066231 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.066291 (+ 60us) server_negotiation.cc:407] Connection header received
0620 20:15:16.068009 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.068033 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.068478 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.068857 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.070958 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.073081 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.075928 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.077539 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.077789 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.085907 (+ 8118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.085922 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.085946 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.086005 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.091005 (+ 5000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.092059 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.092107 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.092124 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.092442 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.092971 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.092977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.092980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.093940 (+ 960us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.094301 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.094884 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.095455 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":427,"threads_started":1}
Jun 20 20:15:16 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:15:16.109122 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.080008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:34807 (local address 127.0.0.1:57946)
0620 20:15:16.081634 (+ 1626us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:16.081692 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:16.081901 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:16.087437 (+ 5536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:16.087448 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:16.088229 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:16.088536 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.088547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.090374 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.090403 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.092365 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.092385 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.093517 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.093525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.093771 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.097737 (+ 3966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:16.097773 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:16.100877 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:16.105721 (+ 4844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.105736 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.105744 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.106477 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.106996 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.107001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.107004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.107170 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.107898 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:16.107906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:16.108106 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 20:15:16.108307 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 20:15:16.108654 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1281,"thread_start_us":1091,"threads_started":1}
I20260620 20:15:16.109136 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.079949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.4.169.190:34807)
0620 20:15:16.086887 (+ 6938us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.086896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.086920 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:15:16.086993 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.087001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.087150 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.087332 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.088728 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.090122 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.092556 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.093274 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.093444 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.101197 (+ 7753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.101206 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.101213 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.101263 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.105515 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.106646 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.106651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.106657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.106860 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.107310 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.107315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.107317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.107731 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.107916 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.108430 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.108655 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6800,"thread_start_us":6685,"threads_started":1}
I20260620 20:15:16.112025 27373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:16 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
Jun 20 20:15:16 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:15:16.124892 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.101122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.4.169.190:34807)
0620 20:15:16.101854 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.101861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.101890 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:15:16.102019 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.102026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.102110 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.102241 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.103942 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.105518 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.108018 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.108924 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.109079 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.116565 (+ 7486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.116573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.116578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.116626 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.120871 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.121576 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.121582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.121585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.121720 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.122149 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.122154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.122157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.122569 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.122850 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.124230 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.124524 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":318,"threads_started":1}
I20260620 20:15:16.125973 27225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } attempt: 4
I20260620 20:15:16.129392 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.112722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:41803 (local address 127.0.0.1:43282)
0620 20:15:16.112892 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:16.112911 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:16.113007 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:16.113536 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:16.113541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:16.113951 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:16.114095 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.114105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.115560 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.115565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.116577 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.116592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.117369 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.117375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.117614 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.118356 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:16.118383 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:16.121606 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:16.125764 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.125772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.125777 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.126314 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.126818 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.126822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.126826 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.127010 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.127904 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:16.127913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:16.128167 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 20:15:16.128381 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 20:15:16.128822 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260620 20:15:16.130806 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.112967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.4.169.189:41803)
0620 20:15:16.113177 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.113183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.113218 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:15:16.113289 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.113294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.113388 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.113498 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.114258 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.115401 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.116727 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.117207 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.117357 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.121912 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.121923 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.121931 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.121996 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.125616 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.126482 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.126488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.126492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.126666 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.127154 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.127159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.127162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.127626 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.127820 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.129779 (+ 1959us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.130099 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:15:16.132186 27373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:16 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
Jun 20 20:15:16 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:15:16.144173 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.126603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54404)
0620 20:15:16.126962 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:16.126982 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:16.127103 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:16.127705 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:16.127712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:16.128100 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:16.128263 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.128272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.129270 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.129276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.130350 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.130361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.131542 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.131548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.131810 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.132520 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:16.132537 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:16.135897 (+ 3360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:16.141236 (+ 5339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.141242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.141247 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.141680 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.142118 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.142123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.142126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.142319 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.143071 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:16.143079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:16.143287 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 20:15:16.143428 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 20:15:16.143743 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260620 20:15:16.144824 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.127281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.4.169.188:39307)
0620 20:15:16.127449 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.127453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.127467 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:15:16.127505 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.127507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.127559 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.127667 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.128406 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.129127 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.130731 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.131372 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.131510 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.136194 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.136205 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.136213 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.136274 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.141082 (+ 4808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.141807 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.141812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.141815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.141989 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.142473 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.142481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.142487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.142917 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.143087 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.143595 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.143912 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:15:16 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
I20260620 20:15:16.148432 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.127141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.4.169.189:41803)
0620 20:15:16.131439 (+ 4298us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.131447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.131470 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:15:16.131659 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.131665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.131736 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.131817 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.133627 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.134650 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.135936 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.136362 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.136499 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.141822 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.141831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.141836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.141879 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.145453 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.146183 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.146189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.146193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.146381 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.146836 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.146842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.146846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.147173 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.147436 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.147782 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.148018 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3688,"thread_start_us":3428,"threads_started":1}
I20260620 20:15:16.151691 27225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567: Bootstrap starting.
I20260620 20:15:16.153885 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:16.154317 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.132974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54420)
0620 20:15:16.133268 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:16.133287 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:16.133383 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:16.139332 (+ 5949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:16.139335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:16.139686 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:16.139925 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.139936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.141695 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.141700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.143021 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.143038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.144000 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.144007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.144250 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.144780 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:16.144795 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:16.147836 (+ 3041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:16.151484 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.151491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.151495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.152005 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.152353 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.152359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.152362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.152515 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.153400 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:16.153405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:16.153586 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 20:15:16.153753 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 20:15:16.154069 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260620 20:15:16.155807 27225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:16.157222 27225 log.cc:824] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:16.157684 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.133118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.4.169.188:39307)
0620 20:15:16.138996 (+ 5878us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.139003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.139030 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:15:16.139085 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.139090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.139172 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.139289 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.140124 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.141558 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.143182 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.143828 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.144541 (+ 713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.148061 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.148069 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.148074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.148109 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.151362 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.152099 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.152106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.152110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.152247 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.152864 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.152869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.152872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.153259 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.153439 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.157036 (+ 3597us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.157286 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5737,"thread_start_us":5568,"threads_started":1}
I20260620 20:15:16.157572 27200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5038613807a545b59916b7f7c3d37567"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57960
I20260620 20:15:16.161684 27225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567: No bootstrap required, opened a new log
I20260620 20:15:16.163385 27373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b: Bootstrap starting.
I20260620 20:15:16.167851 27373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:16.169287 27373 log.cc:824] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:16.169574 27225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } }
I20260620 20:15:16.170071 27225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5038613807a545b59916b7f7c3d37567, State: Initialized, Role: FOLLOWER
I20260620 20:15:16.170645 27225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } }
I20260620 20:15:16.171659 27384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.172179 27384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:16.172878 27373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b: No bootstrap required, opened a new log
I20260620 20:15:16.172922 27225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:16.173014 27200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:15:16.173235 27200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:16.177273 27200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } }
I20260620 20:15:16.180573 27200 leader_election.cc:288] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [CANDIDATE]: Term 1 election: Requested vote from peers ac4f9f471a874097bb6a8365e97d8643 (127.4.169.189:41803), bb57b7797e164cdcbe5a46cbffe8fd3b (127.4.169.188:39307)
I20260620 20:15:16.184398 27373 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } }
I20260620 20:15:16.184975 27373 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb57b7797e164cdcbe5a46cbffe8fd3b, State: Initialized, Role: FOLLOWER
I20260620 20:15:16.185634 27373 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } }
I20260620 20:15:16.188024 27399 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.188944 27399 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:16.189143 27373 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:16.189232 27348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5038613807a545b59916b7f7c3d37567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b"
I20260620 20:15:16.189829 27348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:16.194089 27348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5038613807a545b59916b7f7c3d37567 in term 1.
I20260620 20:15:16.194914 27177 leader_election.cc:302] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5038613807a545b59916b7f7c3d37567, bb57b7797e164cdcbe5a46cbffe8fd3b; no voters:
I20260620 20:15:16.195555 27384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [term 1 FOLLOWER]: Leader election won for term 1
W20260620 20:15:16.197361 27395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:15:16.197593 27395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:15:16.210438 27384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [term 1 LEADER]: Becoming Leader. State: Replica: 5038613807a545b59916b7f7c3d37567, State: Running, Role: LEADER
I20260620 20:15:16.211571 27384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } }
I20260620 20:15:16.188269 27158 master_runner.cc:186] Error getting master registration for 127.4.169.189:41803: OK, instance_id { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" instance_seqno: 1781986515502513 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:15:16.213290 27158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:15:16.223873 27179 tablet.cc:2369] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:16.226971 27401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5038613807a545b59916b7f7c3d37567. Latest consensus state: current_term: 1 leader_uuid: "5038613807a545b59916b7f7c3d37567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.227445 27401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:16.234768 27299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } attempt: 5
I20260620 20:15:16.238034 27408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 20:15:16 dist-test-slave-4bwq krb5kdc[27145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
I20260620 20:15:16.246902 27408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:16.258101 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.235291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:39307 (local address 127.0.0.1:54440)
0620 20:15:16.235645 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:16.235667 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:16.235766 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:16.236596 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:16.236601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:16.237084 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:16.237273 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.237286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.238785 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.238792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.240244 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:16.240259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.241476 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.241484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:16.241748 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.242519 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:16.242567 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:16.245595 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:16.254009 (+ 8414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.254015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.254018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.254625 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.255051 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:16.255058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:16.255062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:16.255226 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:16.257005 (+ 1779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:16.257010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:16.257167 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 20:15:16.257334 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 20:15:16.257668 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260620 20:15:16.259140 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.235977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.4.169.188:39307)
0620 20:15:16.236180 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.236185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.236202 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:15:16.236257 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.236263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.236335 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.236423 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.237450 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.238625 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.240418 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.241123 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.241234 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.245817 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.245826 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.245830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.245868 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.253854 (+ 7986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.254779 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.254786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.254791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.254923 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.255382 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.255389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.255395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.256830 (+ 1435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.257136 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.258618 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.258837 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":724,"server-negotiator.queue_time_us":67}
I20260620 20:15:16.259156 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.221176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.4.169.188:39307)
0620 20:15:16.221357 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.221363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.230031 (+ 8668us) server_negotiation.cc:407] Connection header received
0620 20:15:16.231501 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.231510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.231650 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.231743 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.237682 (+ 5939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.238703 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.240883 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.241530 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.241619 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.249306 (+ 7687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.249316 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.249322 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.249370 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.253116 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.254434 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.254440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.254443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.254652 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.255413 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.255417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.255420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.256033 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.256512 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.258092 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.258560 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:15:16.261927 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.219769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.4.169.190:34807)
0620 20:15:16.220053 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.220063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.225337 (+ 5274us) server_negotiation.cc:407] Connection header received
0620 20:15:16.226219 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.226229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.226352 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.226473 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.234454 (+ 7981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.235546 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.240501 (+ 4955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.241458 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.241630 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.248410 (+ 6780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.248419 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.248429 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.248472 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.253350 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.256369 (+ 3019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.256374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.256377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.256526 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.259704 (+ 3178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.259709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.259712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.260118 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.260379 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.261315 (+ 936us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.261567 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:15:16.265714 27299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643: Bootstrap starting.
I20260620 20:15:16.270486 27299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:16.271909 27299 log.cc:824] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:16.274979 27348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [term 1 FOLLOWER]: Refusing update from remote peer 5038613807a545b59916b7f7c3d37567: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:16.275215 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.230769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.4.169.189:41803)
0620 20:15:16.230947 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.230953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.232352 (+ 1399us) server_negotiation.cc:407] Connection header received
0620 20:15:16.233331 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.233338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.233429 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.233503 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.245636 (+ 12133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.246483 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.252974 (+ 6491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.253933 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.254090 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.262437 (+ 8347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:16.262449 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:16.262453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:16.262498 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:16.268191 (+ 5693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.269204 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.269213 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.269219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.269547 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:16.270721 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:16.270754 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:16.270760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:16.271614 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:16.272199 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.273581 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.274168 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:15:16.276088 27401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:15:16.291853 27327 tablet.cc:2369] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:16.294032 27399 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5038613807a545b59916b7f7c3d37567. Latest consensus state: current_term: 1 leader_uuid: "5038613807a545b59916b7f7c3d37567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.294412 27399 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:16.299029 27401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5038613807a545b59916b7f7c3d37567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.299340 27401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:16.303087 27299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643: No bootstrap required, opened a new log
I20260620 20:15:16.307832 27408 catalog_manager.cc:1347] Generated new cluster ID: 477bea98f64b48b28f1a66f947f91055
I20260620 20:15:16.307955 27408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:16.317873 27399 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5038613807a545b59916b7f7c3d37567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.318282 27399 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:16.328642 27299 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } }
I20260620 20:15:16.332412 27299 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4f9f471a874097bb6a8365e97d8643, State: Initialized, Role: FOLLOWER
I20260620 20:15:16.333000 27299 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } }
I20260620 20:15:16.340001 27426 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.342806 27424 catalog_manager.cc:1259] Loaded cluster ID: 477bea98f64b48b28f1a66f947f91055
I20260620 20:15:16.342904 27424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b: loading cluster ID for follower catalog manager: success
W20260620 20:15:16.344156 27424 catalog_manager.cc:1576] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 20:15:16.344271 27424 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:15:16.344997 27426 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:16.344866 27299 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:16.344950 27274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5038613807a545b59916b7f7c3d37567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac4f9f471a874097bb6a8365e97d8643"
I20260620 20:15:16.345611 27273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:16.349351 27273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [term 1 FOLLOWER]: Refusing update from remote peer 5038613807a545b59916b7f7c3d37567: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:16.352699 27274 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5038613807a545b59916b7f7c3d37567 in term 1.
I20260620 20:15:16.354123 27408 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:16.355127 27401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:16.356323 27408 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:16.334438 27306 master_runner.cc:186] Error getting master registration for 127.4.169.189:41803: OK, instance_id { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" instance_seqno: 1781986515502513 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:15:16.370136 27306 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:15:16.409878 27408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567: Generated new TSK 0
I20260620 20:15:16.411140 27408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:15:16.411729 27426 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5038613807a545b59916b7f7c3d37567. Latest consensus state: current_term: 1 leader_uuid: "5038613807a545b59916b7f7c3d37567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.412051 27426 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:16.412451 27433 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5038613807a545b59916b7f7c3d37567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.412828 27433 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:16.413007 27401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5038613807a545b59916b7f7c3d37567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5038613807a545b59916b7f7c3d37567" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 34807 } } peers { permanent_uuid: "ac4f9f471a874097bb6a8365e97d8643" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 41803 } } peers { permanent_uuid: "bb57b7797e164cdcbe5a46cbffe8fd3b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 39307 } } }
I20260620 20:15:16.413254 27401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5038613807a545b59916b7f7c3d37567 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:16.441679 27443 catalog_manager.cc:1259] Loaded cluster ID: 477bea98f64b48b28f1a66f947f91055
I20260620 20:15:16.441804 27443 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643: loading cluster ID for follower catalog manager: success
I20260620 20:15:16.446039 27443 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643: acquiring CA information for follower catalog manager: success
I20260620 20:15:16.449097 27443 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac4f9f471a874097bb6a8365e97d8643: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 20:15:16.561847 27379 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:16.549518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.4.169.190:34807)
0620 20:15:16.549710 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.549717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.558978 (+ 9261us) server_negotiation.cc:407] Connection header received
0620 20:15:16.560124 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.560134 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.560232 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.560313 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.561540 (+ 1227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57968: BlockingRecv error: recv got EOF from 127.0.0.1:57968 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 20:15:16.568461 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:16.553276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.4.169.188:39307)
0620 20:15:16.553449 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.553455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.563407 (+ 9952us) server_negotiation.cc:407] Connection header received
0620 20:15:16.565358 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.565365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.565477 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.565549 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.568176 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54446: BlockingRecv error: recv got EOF from 127.0.0.1:54446 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 20:15:16.569577 27380 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:16.564292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.4.169.189:41803)
0620 20:15:16.564502 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.564508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.566289 (+ 1781us) server_negotiation.cc:407] Connection header received
0620 20:15:16.566541 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.566547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.566627 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.566697 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.569329 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43308: BlockingRecv error: recv got EOF from 127.0.0.1:43308 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:15:16.618592 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.602158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.4.169.189:41803)
0620 20:15:16.602355 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.602362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.610668 (+ 8306us) server_negotiation.cc:407] Connection header received
0620 20:15:16.611005 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.611012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.611273 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.611415 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:16.612130 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.613349 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.615451 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.616170 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.616276 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.617003 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:16.617725 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:16.617850 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.618103 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.618194 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 20:15:16.662192 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:16.652643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.4.169.188:39307)
0620 20:15:16.652857 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.652864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.659944 (+ 7080us) server_negotiation.cc:407] Connection header received
0620 20:15:16.660540 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.660547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.660687 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.660767 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.661926 (+ 1159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54460: BlockingRecv error: recv got EOF from 127.0.0.1:54460 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:15:16.709199 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.694401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.4.169.189:41803)
0620 20:15:16.694603 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.694611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.701895 (+ 7284us) server_negotiation.cc:407] Connection header received
0620 20:15:16.702414 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.702420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.702516 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.702583 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:16.703279 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.704429 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.706703 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.707450 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.707576 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.708263 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:16.708425 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:16.708489 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.708727 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.708822 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 20:15:16.752024 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:16.742851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.4.169.188:39307)
0620 20:15:16.743074 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.743081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.749997 (+ 6916us) server_negotiation.cc:407] Connection header received
0620 20:15:16.750529 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.750536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.750633 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.750708 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.751745 (+ 1037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54470: BlockingRecv error: recv got EOF from 127.0.0.1:54470 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:15:16.802979 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.787084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.4.169.189:41803)
0620 20:15:16.787301 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.787308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.795895 (+ 8587us) server_negotiation.cc:407] Connection header received
0620 20:15:16.796369 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.796375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.796466 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.796529 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:16.797191 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.798468 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.800560 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.801313 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.801428 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.802072 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:16.802236 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:16.802303 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.802574 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.802668 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 20:15:16.848784 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:16.836447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.4.169.188:39307)
0620 20:15:16.836659 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.836669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.847001 (+ 10332us) server_negotiation.cc:407] Connection header received
0620 20:15:16.847275 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.847283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.847383 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.847465 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.848470 (+ 1005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54482: BlockingRecv error: recv got EOF from 127.0.0.1:54482 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:15:16.910470 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.890477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.4.169.189:41803)
0620 20:15:16.890756 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.890764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.903033 (+ 12269us) server_negotiation.cc:407] Connection header received
0620 20:15:16.903248 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.903255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.903361 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.903464 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:16.904425 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.905634 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.908167 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.908785 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.908892 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.909556 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:16.909723 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:16.909792 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.910025 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.910119 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
W20260620 20:15:16.960021 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:16.950974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.4.169.188:39307)
0620 20:15:16.951225 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.951231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.958089 (+ 6858us) server_negotiation.cc:407] Connection header received
0620 20:15:16.958592 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.958599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.958697 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.958767 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.959770 (+ 1003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54492: BlockingRecv error: recv got EOF from 127.0.0.1:54492 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 20:15:16.977965 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:16.971974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.4.169.189:41803)
0620 20:15:16.972198 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.972205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.972780 (+ 575us) server_negotiation.cc:407] Connection header received
0620 20:15:16.973010 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.973019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.973130 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.973243 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:16.973690 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.974691 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.976108 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:16.976570 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:16.976707 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:16.977107 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:16.977238 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:16.977336 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:16.977482 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 20:15:16.977570 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 20:15:16.986734 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:16.984450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.4.169.188:39307)
0620 20:15:16.984620 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:16.984627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:16.985298 (+ 671us) server_negotiation.cc:407] Connection header received
0620 20:15:16.985525 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:16.985533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:16.985615 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:16.985723 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:16.986462 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54500: BlockingRecv error: recv got EOF from 127.0.0.1:54500 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:15:17.025566 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:17.020141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.4.169.189:41803)
0620 20:15:17.020373 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.020379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.020946 (+ 567us) server_negotiation.cc:407] Connection header received
0620 20:15:17.021194 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.021198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.021266 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.021369 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:17.021798 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.022603 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.023888 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.024324 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.024461 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:17.024868 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:17.024967 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:17.025067 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:17.025211 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 20:15:17.025290 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260620 20:15:17.038076 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:17.036068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.4.169.188:39307)
0620 20:15:17.036293 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.036302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.036656 (+ 354us) server_negotiation.cc:407] Connection header received
0620 20:15:17.036879 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.036886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.036991 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.037093 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:17.037848 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54504: BlockingRecv error: recv got EOF from 127.0.0.1:54504 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 20:15:17.079771 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:17.073532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.4.169.189:41803)
0620 20:15:17.073715 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.073721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.074288 (+ 567us) server_negotiation.cc:407] Connection header received
0620 20:15:17.074506 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.074510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.074584 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.074666 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:17.075175 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.075983 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.077933 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.078455 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.078615 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:17.079025 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:17.079180 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:17.079249 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:17.079393 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 20:15:17.079460 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 20:15:17.087795 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:17.085735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.4.169.188:39307)
0620 20:15:17.085900 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.085905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.086569 (+ 664us) server_negotiation.cc:407] Connection header received
0620 20:15:17.086720 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.086724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.086797 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.086880 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:17.087545 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54508: BlockingRecv error: recv got EOF from 127.0.0.1:54508 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:15:17.160567 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:17.155143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.4.169.189:41803)
0620 20:15:17.155433 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.155440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.155725 (+ 285us) server_negotiation.cc:407] Connection header received
0620 20:15:17.155951 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.155961 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.156029 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.156161 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:17.156624 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.157393 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.158861 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.159292 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.159430 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:17.159827 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:17.159937 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:17.160035 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:17.160197 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 20:15:17.160276 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260620 20:15:17.168298 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:17.166223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.4.169.188:39307)
0620 20:15:17.166476 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.166483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.166720 (+ 237us) server_negotiation.cc:407] Connection header received
0620 20:15:17.166940 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.166946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.167098 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.167214 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:17.168024 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54510: BlockingRecv error: recv got EOF from 127.0.0.1:54510 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:15:17.304136 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:17.299495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.4.169.189:41803)
0620 20:15:17.299754 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.299761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.299940 (+ 179us) server_negotiation.cc:407] Connection header received
0620 20:15:17.300083 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.300087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.300167 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.300250 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:17.300609 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.301359 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.302646 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.303038 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.303127 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:17.303475 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:17.303570 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:17.303633 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:17.303754 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 20:15:17.303815 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260620 20:15:17.311596 27376 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:17.309758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.4.169.188:39307)
0620 20:15:17.309978 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.309985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.310249 (+ 264us) server_negotiation.cc:407] Connection header received
0620 20:15:17.310471 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.310475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.310545 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.310665 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:17.311397 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54516: BlockingRecv error: recv got EOF from 127.0.0.1:54516 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:15:17.347987 27424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b: acquiring CA information for follower catalog manager: success
I20260620 20:15:17.349808 27424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb57b7797e164cdcbe5a46cbffe8fd3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:15:17.576054 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:17.571152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.4.169.189:41803)
0620 20:15:17.571391 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.571397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.571653 (+ 256us) server_negotiation.cc:407] Connection header received
0620 20:15:17.571820 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.571824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.571908 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.571988 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:17.572361 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.573062 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.574539 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.574922 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.575018 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:17.575363 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:17.575459 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:17.575541 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:17.575655 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 20:15:17.575717 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 20:15:17.587061 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:17.581506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.4.169.188:39307)
0620 20:15:17.581707 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.581713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.581862 (+ 149us) server_negotiation.cc:407] Connection header received
0620 20:15:17.582071 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.582076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.582243 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.582367 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:17.582836 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.583497 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.584786 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.585221 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.585357 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:17.585826 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:17.586379 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:17.586592 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:17.586695 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 20:15:17.586776 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:15:17.601723 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:17.593428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.4.169.188:39307)
0620 20:15:17.593647 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.593656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.595300 (+ 1644us) server_negotiation.cc:407] Connection header received
0620 20:15:17.595517 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.595525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.595626 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.595721 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:17.596496 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.597365 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.599547 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.600175 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.600278 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:17.600860 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:17.600988 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:17.601086 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:17.601296 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 20:15:17.601376 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:15:17.602017 27558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:17.593363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.4.169.190:34807)
0620 20:15:17.595461 (+ 2098us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.595471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.595497 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:15:17.595639 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.595645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.595880 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.596072 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:17.596730 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.597773 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.599486 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.600114 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.600266 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:17.600816 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:17.601224 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:17.601319 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:17.601652 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:15:17.601722 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1923,"thread_start_us":1792,"threads_started":1}
I20260620 20:15:17.603315 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:17.594331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.4.169.189:41803)
0620 20:15:17.594609 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:17.594617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:17.596008 (+ 1391us) server_negotiation.cc:407] Connection header received
0620 20:15:17.596410 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:17.596422 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:17.596522 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:17.596615 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:17.597194 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.598067 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.599978 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:17.600553 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:17.600690 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:17.601267 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:17.601385 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:17.601466 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:17.603028 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0620 20:15:17.603115 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 20:15:17.607287 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 27158
I20260620 20:15:17.608367 27220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:17.720198 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 27158
I20260620 20:15:17.740691 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 27232
I20260620 20:15:17.741813 27294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:17.850240 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 27232
I20260620 20:15:17.870060 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 27306
I20260620 20:15:17.871243 27368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:17.977006 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 27306
2026-06-20T20:15:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4793 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:18 dist-test-slave-4bwq krb5kdc[27566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:18 dist-test-slave-4bwq krb5kdc[27566](info): set up 2 sockets
Jun 20 20:15:18 dist-test-slave-4bwq krb5kdc[27566](info): commencing operation
krb5kdc: starting...
W20260620 20:15:19.115442 4774 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/test-user.keytab.
Jun 20 20:15:19 dist-test-slave-4bwq krb5kdc[27566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.169.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
Entry for principal kudu/127.4.169.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab.
I20260620 20:15:19.369454 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42601
--webserver_interface=127.4.169.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:42601,127.4.169.189:42391,127.4.169.188:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:19.704571 27579 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:15:19.704957 27579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:19.705101 27579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:19.705240 27579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:19.712760 27579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:19.712867 27579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:19.712978 27579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:19.713081 27579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:19.713189 27579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:19.723505 27579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:42601,127.4.169.189:42391,127.4.169.188:46715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.190.keytab
--principal=kudu/127.4.169.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.190:42601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.169.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:19.724911 27579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:19.726279 27579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:19.738240 27585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:19.738312 27587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:19.738267 27584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:19.738996 27579 server_base.cc:1034] running on GCE node
I20260620 20:15:19.739825 27579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:19.741258 27579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:19.742511 27579 hybrid_clock.cc:648] HybridClock initialized: now 1781986519742446 us; error 31 us; skew 500 ppm
I20260620 20:15:19.742871 27579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:19 dist-test-slave-4bwq krb5kdc[27566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986519, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:19.746059 27579 init.cc:377] Logged in from keytab as kudu/127.4.169.190@KRBTEST.COM (short username kudu)
I20260620 20:15:19.747663 27579 webserver.cc:460] Webserver started at http://127.4.169.190:37917/ using document root <none> and password file <none>
I20260620 20:15:19.748332 27579 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:19.748420 27579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:19.748682 27579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:19.751729 27579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/data/instance:
uuid: "ddef03c3a0df4b469a15c52b10c55069"
format_stamp: "Formatted at 2026-06-20 20:15:19 on dist-test-slave-4bwq"
server_key: "c94c966bda6c6f4a7fda9a22a0eaa12c"
server_key_iv: "6f43c7b1d22edcdf6dab6ab3e2653a92"
server_key_version: "encryptionkey@0"
I20260620 20:15:19.752573 27579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/wal/instance:
uuid: "ddef03c3a0df4b469a15c52b10c55069"
format_stamp: "Formatted at 2026-06-20 20:15:19 on dist-test-slave-4bwq"
server_key: "c94c966bda6c6f4a7fda9a22a0eaa12c"
server_key_iv: "6f43c7b1d22edcdf6dab6ab3e2653a92"
server_key_version: "encryptionkey@0"
I20260620 20:15:19.757732 27579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:15:19.761171 27594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:19.762633 27579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:15:19.762817 27579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/wal
uuid: "ddef03c3a0df4b469a15c52b10c55069"
format_stamp: "Formatted at 2026-06-20 20:15:19 on dist-test-slave-4bwq"
server_key: "c94c966bda6c6f4a7fda9a22a0eaa12c"
server_key_iv: "6f43c7b1d22edcdf6dab6ab3e2653a92"
server_key_version: "encryptionkey@0"
I20260620 20:15:19.763041 27579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:19.776829 27579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:19.780961 27579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:19.781260 27579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:19.800742 27579 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.190:42601
I20260620 20:15:19.800761 27645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.190:42601 every 8 connection(s)
I20260620 20:15:19.802138 27579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:19.803434 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 27579
I20260620 20:15:19.803620 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-0/wal/instance
I20260620 20:15:19.804142 4774 external_mini_cluster.cc:1428] Setting key e366bc41f046456055f0b0088ac08b06
I20260620 20:15:19.806810 27646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:19.816442 27646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.169.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.189@KRBTEST.COM" created.
Jun 20 20:15:19 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986519, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:15:19.845012 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:19.819735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.4.169.190:42601)
0620 20:15:19.820488 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:19.820498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:19.827120 (+ 6622us) server_negotiation.cc:407] Connection header received
0620 20:15:19.829197 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:19.829204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:19.829440 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:19.829650 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:19.831011 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:19.832195 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:19.833563 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:19.834173 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:19.834370 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:19.837833 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:19.837841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:19.837852 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:19.837879 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:19.840652 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:19.841396 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:19.841402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:19.841404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:19.841537 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:19.841921 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:19.841927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:19.841935 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:19.842458 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:19.842662 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:19.843193 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 20:15:19.843572 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":570,"thread_start_us":351,"threads_started":1}
I20260620 20:15:19.845036 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:19.819742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42601 (local address 127.0.0.1:35402)
0620 20:15:19.826924 (+ 7182us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:19.826965 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:19.828822 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:19.829658 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:19.829665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:19.830459 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:19.830796 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:19.830814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:19.832352 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:19.832357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:19.833427 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:19.833438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:19.834423 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:19.834430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:19.834601 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:19.835276 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:19.835304 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:19.837607 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:19.840796 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:19.840808 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:19.840820 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:19.841255 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:19.841641 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:19.841648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:19.841651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:19.841805 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:19.842593 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:19.842603 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:19.842827 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 20:15:19.843065 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 20:15:19.843686 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6956,"mutex_wait_us":159,"thread_start_us":6774,"threads_started":1}
I20260620 20:15:19.848037 27646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:19.849120 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:19.848688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51432)
0620 20:15:19.848883 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:19.848990 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260620 20:15:19.849475 27596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
I20260620 20:15:19.851018 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:19.850704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51436)
0620 20:15:19.850834 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:19.850900 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:15:19.851310 27646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:42391: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab.
Entry for principal kudu/127.4.169.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab.
I20260620 20:15:19.865461 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.189:42391
--webserver_interface=127.4.169.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab
--principal=kudu/127.4.169.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:42601,127.4.169.189:42391,127.4.169.188:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:15:19.900660 27646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } attempt: 1
I20260620 20:15:19.901845 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:19.901417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51448)
0620 20:15:19.901608 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:19.901710 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 20:15:19.902829 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:19.902515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51454)
0620 20:15:19.902642 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:19.902708 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:15:19.903030 27646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:42391: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
I20260620 20:15:19.971275 27646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } attempt: 2
I20260620 20:15:19.972479 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:19.972004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51456)
0620 20:15:19.972256 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:19.972345 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 20:15:19.973498 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:19.973168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51466)
0620 20:15:19.973302 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:19.973378 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:15:19.973742 27646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:42391: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
I20260620 20:15:20.065008 27646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } attempt: 3
I20260620 20:15:20.066365 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.065809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51478)
0620 20:15:20.066055 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.066188 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 20:15:20.067561 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.067136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51488)
0620 20:15:20.067292 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.067380 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:15:20.067819 27646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:42391: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
W20260620 20:15:20.198650 27653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:15:20.199007 27653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:20.199155 27653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:20.199290 27653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:20.206496 27653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:20.206583 27653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:20.206732 27653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:20.206827 27653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:20.206920 27653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:20.211063 27646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } attempt: 4
I20260620 20:15:20.212343 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.211801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51492)
0620 20:15:20.212102 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.212191 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 20:15:20.213341 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.213000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51498)
0620 20:15:20.213154 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.213235 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:15:20.213557 27646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.189:42391: Network error: Client connection negotiation failed: client connection to 127.4.169.189:42391: connect: Connection refused (error 111)
I20260620 20:15:20.217450 27653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:42601,127.4.169.189:42391,127.4.169.188:46715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.189.keytab
--principal=kudu/127.4.169.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.189:42391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.169.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:20.218870 27653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:20.220176 27653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:20.231925 27659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:20.231995 27661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:20.232143 27658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:20.232913 27653 server_base.cc:1034] running on GCE node
I20260620 20:15:20.233675 27653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:20.235123 27653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:20.236348 27653 hybrid_clock.cc:648] HybridClock initialized: now 1781986520236284 us; error 34 us; skew 500 ppm
I20260620 20:15:20.236675 27653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986520, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:20.240023 27653 init.cc:377] Logged in from keytab as kudu/127.4.169.189@KRBTEST.COM (short username kudu)
I20260620 20:15:20.241567 27653 webserver.cc:460] Webserver started at http://127.4.169.189:45455/ using document root <none> and password file <none>
I20260620 20:15:20.242264 27653 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:20.242379 27653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:20.242810 27653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:20.245795 27653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/data/instance:
uuid: "86740971fc184137a86d2946168911d7"
format_stamp: "Formatted at 2026-06-20 20:15:20 on dist-test-slave-4bwq"
server_key: "c89293a8fd23c616c7b3cade09f2884a"
server_key_iv: "cdd0cab6c5354638c066b0d66e70bcc8"
server_key_version: "encryptionkey@0"
I20260620 20:15:20.246670 27653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/wal/instance:
uuid: "86740971fc184137a86d2946168911d7"
format_stamp: "Formatted at 2026-06-20 20:15:20 on dist-test-slave-4bwq"
server_key: "c89293a8fd23c616c7b3cade09f2884a"
server_key_iv: "cdd0cab6c5354638c066b0d66e70bcc8"
server_key_version: "encryptionkey@0"
I20260620 20:15:20.251767 27653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 20:15:20.255101 27668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:20.256436 27653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:15:20.256611 27653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/wal
uuid: "86740971fc184137a86d2946168911d7"
format_stamp: "Formatted at 2026-06-20 20:15:20 on dist-test-slave-4bwq"
server_key: "c89293a8fd23c616c7b3cade09f2884a"
server_key_iv: "cdd0cab6c5354638c066b0d66e70bcc8"
server_key_version: "encryptionkey@0"
I20260620 20:15:20.256794 27653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:20.282660 27653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:20.286695 27653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:20.286968 27653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:20.305522 27653 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.189:42391
I20260620 20:15:20.305546 27719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.189:42391 every 8 connection(s)
I20260620 20:15:20.306913 27653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/data/info.pb
I20260620 20:15:20.309902 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 27653
I20260620 20:15:20.310081 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-1/wal/instance
I20260620 20:15:20.310571 4774 external_mini_cluster.cc:1428] Setting key e2b8b982d709ec3ced99e0f423d8a260
I20260620 20:15:20.310707 27720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:20.317286 27720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986520, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.169.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.169.188@KRBTEST.COM" created.
I20260620 20:15:20.339566 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.320332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.4.169.190:42601)
0620 20:15:20.320566 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.320574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.321889 (+ 1315us) server_negotiation.cc:407] Connection header received
0620 20:15:20.323827 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.323836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.323935 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.324053 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.325524 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.326669 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.328208 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.328934 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.329073 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.332926 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.332933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.332938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.332983 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.336429 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.337120 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.337123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.337125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.337293 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.337687 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.337693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.337697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.338125 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.338279 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.339164 (+ 885us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.339332 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:15:20.341131 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.320853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42601 (local address 127.0.0.1:35412)
0620 20:15:20.321705 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.321757 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:20.323556 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:20.324277 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:20.324284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:20.324801 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:20.325349 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.325364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.326839 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.326843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.328048 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.328060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.329157 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.329162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.329360 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.330078 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:20.330101 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:20.332700 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:20.336580 (+ 3880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.336587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.336599 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.337015 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.337405 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.337409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.337411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.337578 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.338263 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:20.338289 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:20.338784 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 20:15:20.339093 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 20:15:20.339657 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":402,"threads_started":1}
I20260620 20:15:20.342792 27720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986520, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:15:20.359283 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.343725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.4.169.189:42391)
0620 20:15:20.344322 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.344327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.344350 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:15:20.344391 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.344397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.344557 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.344707 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.345432 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.346513 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.347637 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.348155 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.348296 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.351775 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.351785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.351797 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.351872 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.356028 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.356906 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.356916 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.356921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.357075 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.357520 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.357525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.357529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.357995 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.358229 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.358717 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.358977 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":329,"threads_started":1}
I20260620 20:15:20.359328 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.343414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:51512)
0620 20:15:20.343617 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.343632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:20.343717 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:20.344697 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:20.344701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:20.345135 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:20.345258 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.345266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.346649 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.346655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.347505 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.347516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.348299 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.348305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.348560 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.349164 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:20.349180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:20.351580 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:20.356161 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.356181 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.356188 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.356752 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.357191 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.357198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.357204 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.357396 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.358149 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:20.358156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:20.358329 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 20:15:20.358501 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 20:15:20.358825 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":56}
I20260620 20:15:20.362020 27720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:20.363257 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.362644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:36348)
0620 20:15:20.362896 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.363038 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 20:15:20.363497 27670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
I20260620 20:15:20.365096 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.364671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:36358)
0620 20:15:20.364870 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.364954 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 20:15:20.365442 27720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:46715: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.169.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab.
Entry for principal kudu/127.4.169.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab.
I20260620 20:15:20.373566 4774 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
/tmp/dist-test-taskBzMLvU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.188:46715
--webserver_interface=127.4.169.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab
--principal=kudu/127.4.169.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.169.148:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.169.190:42601,127.4.169.189:42391,127.4.169.188:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:15:20.414765 27720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } attempt: 1
I20260620 20:15:20.415943 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.415499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:36364)
0620 20:15:20.415684 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.415791 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 20:15:20.417061 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.416706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:36380)
0620 20:15:20.416886 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.416962 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 20:15:20.417301 27720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:46715: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
I20260620 20:15:20.485524 27720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } attempt: 2
I20260620 20:15:20.486737 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.486234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46686)
0620 20:15:20.486469 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.486566 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 20:15:20.487982 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.487598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46700)
0620 20:15:20.487787 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.487859 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:15:20.488271 27720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:46715: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
I20260620 20:15:20.512821 27646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } attempt: 5
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986519, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:15:20.527935 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.513559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:55076)
0620 20:15:20.513804 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.513828 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:20.513965 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:20.514340 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:20.514345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:20.514876 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:20.515175 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.515192 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.516191 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.516196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.517756 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.517771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.518474 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.518480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.518706 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.519369 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:20.519392 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:20.521770 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:20.525676 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.525681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.525683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.526064 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.526473 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.526477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.526479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.526608 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.527220 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:20.527223 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:20.527349 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 20:15:20.527465 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 20:15:20.527667 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 20:15:20.528167 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.513733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.4.169.189:42391)
0620 20:15:20.513891 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.513895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.513915 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:15:20.514094 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.514097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.514179 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.514268 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.515343 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.516066 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.517901 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.518304 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.518460 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.521999 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.522008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.522014 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.522065 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.525498 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.526182 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.526187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.526191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.526338 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.526711 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.526716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.526719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.527093 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.527239 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.527597 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.527861 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:15:20.529198 27646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:20.530241 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.529863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46710)
0620 20:15:20.530025 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.530099 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:15:20.531195 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.530868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46720)
0620 20:15:20.531019 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.531097 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:15:20.531388 27646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:46715: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
I20260620 20:15:20.579540 27720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } attempt: 3
I20260620 20:15:20.580833 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.580248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46736)
0620 20:15:20.580485 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.580629 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 20:15:20.582016 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.581543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46752)
0620 20:15:20.581787 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.581872 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260620 20:15:20.582263 27720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:46715: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
I20260620 20:15:20.582576 27646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } attempt: 1
I20260620 20:15:20.583550 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.583150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46754)
0620 20:15:20.583323 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.583406 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 20:15:20.584677 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.584283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46762)
0620 20:15:20.584443 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.584524 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:15:20.584923 27646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:46715: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
I20260620 20:15:20.653175 27646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } attempt: 2
I20260620 20:15:20.654497 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.653895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46776)
0620 20:15:20.654147 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.654259 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 20:15:20.655676 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.655239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46780)
0620 20:15:20.655400 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.655511 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:15:20.655952 27646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:46715: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
W20260620 20:15:20.712237 27727 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:15:20.712597 27727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:20.712724 27727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:20.712818 27727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:20.720504 27727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:20.720594 27727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:20.720695 27727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:20.720780 27727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:20.720888 27727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:20.725509 27720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } attempt: 4
I20260620 20:15:20.726725 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.726128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46790)
0620 20:15:20.726379 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.726516 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 20:15:20.727749 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.727389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46796)
0620 20:15:20.727537 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.727616 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:15:20.727949 27720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:46715: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
I20260620 20:15:20.731566 27727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.169.148:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.169.190:42601,127.4.169.189:42391,127.4.169.188:46715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/krb5kdc/kudu_127.4.169.188.keytab
--principal=kudu/127.4.169.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.169.188:46715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.169.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:20.732846 27727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:20.734277 27727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:20.745669 27732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:20.745771 27735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:20.745771 27733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:20.746891 27727 server_base.cc:1034] running on GCE node
I20260620 20:15:20.747678 27727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:20.749086 27727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:20.750334 27727 hybrid_clock.cc:648] HybridClock initialized: now 1781986520750258 us; error 54 us; skew 500 ppm
I20260620 20:15:20.750717 27727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986520, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:20.753820 27727 init.cc:377] Logged in from keytab as kudu/127.4.169.188@KRBTEST.COM (short username kudu)
I20260620 20:15:20.755489 27727 webserver.cc:460] Webserver started at http://127.4.169.188:38187/ using document root <none> and password file <none>
I20260620 20:15:20.756183 27727 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:20.756320 27727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:20.756639 27727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:20.759609 27727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/data/instance:
uuid: "0cd761c4e3664090a3b30e69ef7cc87b"
format_stamp: "Formatted at 2026-06-20 20:15:20 on dist-test-slave-4bwq"
server_key: "58bb78f20e38d7a220ff8df3289869d2"
server_key_iv: "095ca3d6fda0593daeb32f577e843fcb"
server_key_version: "encryptionkey@0"
I20260620 20:15:20.760414 27727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/wal/instance:
uuid: "0cd761c4e3664090a3b30e69ef7cc87b"
format_stamp: "Formatted at 2026-06-20 20:15:20 on dist-test-slave-4bwq"
server_key: "58bb78f20e38d7a220ff8df3289869d2"
server_key_iv: "095ca3d6fda0593daeb32f577e843fcb"
server_key_version: "encryptionkey@0"
I20260620 20:15:20.762187 27646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } attempt: 3
I20260620 20:15:20.763278 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.762833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46800)
0620 20:15:20.763039 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.763123 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:15:20.764252 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.763924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46808)
0620 20:15:20.764075 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.764154 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:15:20.764510 27646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.169.188:46715: Network error: Client connection negotiation failed: client connection to 127.4.169.188:46715: connect: Connection refused (error 111)
I20260620 20:15:20.765861 27727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:15:20.769452 27742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:20.770843 27727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:20.771021 27727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/data,/tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/wal
uuid: "0cd761c4e3664090a3b30e69ef7cc87b"
format_stamp: "Formatted at 2026-06-20 20:15:20 on dist-test-slave-4bwq"
server_key: "58bb78f20e38d7a220ff8df3289869d2"
server_key_iv: "095ca3d6fda0593daeb32f577e843fcb"
server_key_version: "encryptionkey@0"
I20260620 20:15:20.771256 27727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:20.791909 27727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:20.795686 27727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:20.795949 27727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:20.816220 27727 rpc_server.cc:307] RPC server started. Bound to: 127.4.169.188:46715
I20260620 20:15:20.816233 27793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.169.188:46715 every 8 connection(s)
I20260620 20:15:20.817523 27727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/data/info.pb
I20260620 20:15:20.818132 4774 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu as pid 27727
I20260620 20:15:20.818303 4774 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBzMLvU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986152907614-4774-0/minicluster-data/master-2/wal/instance
I20260620 20:15:20.819067 4774 external_mini_cluster.cc:1428] Setting key 729152d82412fd880ad5a7d902b243f8
I20260620 20:15:20.822021 27794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:20.828763 27794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986520, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
I20260620 20:15:20.841615 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.820828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.4.169.188:46715)
0620 20:15:20.821515 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.821526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.821565 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:15:20.823265 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.823285 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.823636 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.823995 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.825362 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.826747 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.827999 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.828639 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.828830 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.833535 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.833551 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.833568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.833620 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.837130 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.837882 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.837887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.837893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.838092 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.838601 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.838606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.838609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.839266 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.839535 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.839946 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.840438 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":304,"threads_started":1}
I20260620 20:15:20.847101 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.831338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.190:42601 (local address 127.0.0.1:56984)
0620 20:15:20.832105 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.832150 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:20.832319 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:20.832712 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:20.832719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:20.833318 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:20.833711 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.833722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.834918 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.834926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.836695 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.836710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.837502 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.837506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.837620 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.838141 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:20.838167 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:20.840755 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:20.844317 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.844328 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.844334 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.844925 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.845324 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.845327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.845329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.845512 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.846028 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:20.846036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:20.846256 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 20:15:20.846501 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 20:15:20.846816 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":336,"threads_started":1}
I20260620 20:15:20.847237 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.831321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.4.169.190:42601)
0620 20:15:20.831499 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.831504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.832262 (+ 758us) server_negotiation.cc:407] Connection header received
0620 20:15:20.832453 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.832460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.832545 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.832654 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.833868 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.834761 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.836844 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.837336 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.837465 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.840968 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.840973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.840976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.841009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.844167 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.845029 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.845035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.845038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.845219 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.845631 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.845635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.845637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.845884 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.846007 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.846575 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.846779 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:15:20.848621 27794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.190@KRBTEST.COM
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986520, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
I20260620 20:15:20.860704 27799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.844472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.4.169.190:42601)
0620 20:15:20.845127 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.845135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.845158 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:15:20.845200 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.845205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.845276 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.845379 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.846827 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.848059 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.849580 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.850197 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.850323 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.854804 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.854809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.854812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.854840 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.858059 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.858785 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.858791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.858795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.858912 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.859265 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.859270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.859275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.859528 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.859793 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.860076 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.860289 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":298,"threads_started":1}
I20260620 20:15:20.862536 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.849252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.189:42391 (local address 127.0.0.1:55082)
0620 20:15:20.849462 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.849483 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:20.849568 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:20.849856 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:20.849860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:20.850311 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:20.850511 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.850519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.851908 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.851911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.852770 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.852780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.853596 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.853602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.853806 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.854315 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:20.854338 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:20.856658 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:20.859982 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.859987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.859991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.860337 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.860752 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.860757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.860761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.860949 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.861639 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:20.861647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:20.861821 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 20:15:20.861954 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 20:15:20.862170 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 20:15:20.862739 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.849385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.4.169.189:42391)
0620 20:15:20.849538 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.849542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.849557 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:15:20.849670 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.849673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.849734 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.849810 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.850666 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.851767 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.852920 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.853444 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.853538 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.856844 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.856852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.856859 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.856900 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.859771 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.860473 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.860480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.860483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.860644 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.861108 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.861113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.861117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.861487 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.861651 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.862072 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.862303 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:15:20.864030 27794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.169.189@KRBTEST.COM
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986520, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.188@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
I20260620 20:15:20.879153 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.864790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46822)
0620 20:15:20.864972 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.864994 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:20.865107 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:20.865450 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:20.865458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:20.866079 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:20.866289 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.866304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.867670 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.867686 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.869547 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.869566 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.870667 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.870683 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.870942 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.871593 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:20.871610 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:20.873446 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:20.876721 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.876728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.876734 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.877162 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.877497 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.877505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.877511 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.877666 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.878135 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:20.878140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:20.878283 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 20:15:20.878468 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 20:15:20.878754 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 20:15:20.879510 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.865005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.4.169.188:46715)
0620 20:15:20.865158 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.865164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.865181 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:15:20.865236 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.865241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.865306 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.865399 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.866476 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.867532 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.869723 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.870486 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.870625 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.873609 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.873613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.873616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.873642 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.876581 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.877272 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.877277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.877279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.877392 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.877779 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.877784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.877787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.878031 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.878171 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.878544 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.878905 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":53}
I20260620 20:15:20.879531 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.862495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.4.169.189:42391)
0620 20:15:20.863349 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.863357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.863380 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:15:20.863461 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.863468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.863557 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.863691 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.865003 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.865965 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.867701 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.868165 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.868305 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.873732 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.873742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.873749 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.873799 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.876889 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.877562 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.877565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.877567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.877706 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.878099 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.878104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.878107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.878371 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.878672 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.878964 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.879175 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":12056,"threads_started":1}
I20260620 20:15:20.884027 4774 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:20.884404 27794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b: Bootstrap starting.
I20260620 20:15:20.887511 27794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:20.888477 27794 log.cc:824] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:20.886255 27621 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ddef03c3a0df4b469a15c52b10c55069"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56994
I20260620 20:15:20.891108 27794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b: No bootstrap required, opened a new log
I20260620 20:15:20.897754 27794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } }
I20260620 20:15:20.898068 27794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd761c4e3664090a3b30e69ef7cc87b, State: Initialized, Role: FOLLOWER
I20260620 20:15:20.898527 27794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } }
I20260620 20:15:20.899416 27802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:20.899801 27802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:20.900269 27794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:15:20.909714 27813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:15:20.909840 27813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:15:20.907377 27727 master_runner.cc:186] Error getting master registration for 127.4.169.190:42601: OK, instance_id { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" instance_seqno: 1781986519794581 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:15:20.918434 27727 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:15:20.941753 27646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } attempt: 4
Jun 20 20:15:20 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986519, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.190@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
I20260620 20:15:20.958731 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.942656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.4.169.188:46715)
0620 20:15:20.942895 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:20.942903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:20.942926 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:15:20.942992 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:20.942997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:20.943073 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:20.943179 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:20.944151 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.945262 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.947168 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.947900 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.948018 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.952017 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:20.952029 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:20.952033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:20.952061 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:20.955692 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.956501 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.956507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.956510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.956654 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:20.957067 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:20.957072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:20.957075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:20.957513 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:20.957665 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:20.958115 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 20:15:20.958336 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:15:20.958729 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:20.942448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46830)
0620 20:15:20.942695 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:20.942719 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:20.942847 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:20.943240 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:20.943245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:20.943792 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:20.944006 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.944018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.945404 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.945410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.946996 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:20.947013 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:20.948053 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:20.948060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:20.948271 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:20.948949 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:20.948975 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:20.951832 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:20.955835 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.955841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.955846 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.956371 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.956771 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:20.956776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:20.956780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:20.956950 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:20.957639 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:20.957644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:20.957844 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 20:15:20.957995 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 20:15:20.958279 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 20:15:20.964706 27646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069: Bootstrap starting.
I20260620 20:15:20.968389 27646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:20.969792 27646 log.cc:824] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:20.972548 27646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069: No bootstrap required, opened a new log
I20260620 20:15:20.978979 27646 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } }
I20260620 20:15:20.979373 27646 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddef03c3a0df4b469a15c52b10c55069, State: Initialized, Role: FOLLOWER
I20260620 20:15:20.979836 27646 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } }
I20260620 20:15:20.980748 27816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:20.981215 27816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:20.981956 27646 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:20.982091 27621 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:15:20.982295 27621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:20.985656 27621 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } }
I20260620 20:15:20.987241 27621 leader_election.cc:288] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [CANDIDATE]: Term 1 election: Requested vote from peers 86740971fc184137a86d2946168911d7 (127.4.169.189:42391), 0cd761c4e3664090a3b30e69ef7cc87b (127.4.169.188:46715)
I20260620 20:15:20.988777 27769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddef03c3a0df4b469a15c52b10c55069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0cd761c4e3664090a3b30e69ef7cc87b"
I20260620 20:15:20.989368 27769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:20.993698 27769 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddef03c3a0df4b469a15c52b10c55069 in term 1.
I20260620 20:15:20.994743 27596 leader_election.cc:302] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cd761c4e3664090a3b30e69ef7cc87b, ddef03c3a0df4b469a15c52b10c55069; no voters:
I20260620 20:15:20.995349 27816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:21.006534 27816 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [term 1 LEADER]: Becoming Leader. State: Replica: ddef03c3a0df4b469a15c52b10c55069, State: Running, Role: LEADER
I20260620 20:15:21.007440 27816 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } }
I20260620 20:15:21.015617 27823 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddef03c3a0df4b469a15c52b10c55069. Latest consensus state: current_term: 1 leader_uuid: "ddef03c3a0df4b469a15c52b10c55069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:21.016217 27823 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:21.027141 27720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } attempt: 5
I20260620 20:15:21.029867 27829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 20:15:21.035065 27600 tablet.cc:2369] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:15:21.048128 27748 tablet.cc:2369] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:15:21 dist-test-slave-4bwq krb5kdc[27566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986520, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.169.189@KRBTEST.COM for kudu/127.4.169.188@KRBTEST.COM
I20260620 20:15:21.052070 27829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:21.064792 27799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:21.008450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.4.169.190:42601)
0620 20:15:21.008702 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.008710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.017645 (+ 8935us) server_negotiation.cc:407] Connection header received
0620 20:15:21.017941 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.017949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.018035 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.018110 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.025265 (+ 7155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.026423 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.037722 (+ 11299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.038445 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.038557 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:21.054544 (+ 15987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:21.054550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:21.054555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:21.054607 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:21.061483 (+ 6876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:21.062365 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:21.062371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:21.062374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:21.062570 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:21.062992 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:21.062997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:21.063001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:21.063343 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:21.063574 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:21.064164 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 20:15:21.064405 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:15:21.065791 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:21.016739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.4.169.189:42391)
0620 20:15:21.016938 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.016944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.021224 (+ 4280us) server_negotiation.cc:407] Connection header received
0620 20:15:21.022348 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.022355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.022461 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.022535 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.032922 (+ 10387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.034266 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.042225 (+ 7959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.042969 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.043059 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:21.055537 (+ 12478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:21.055544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:21.055550 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:21.055605 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:21.059252 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:21.061004 (+ 1752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:21.061014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:21.061020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:21.061211 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:21.062107 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:21.062116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:21.062122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:21.062901 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:21.063382 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:21.064711 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0620 20:15:21.065111 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:15:21.080425 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:21.021711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.4.169.188:46715)
0620 20:15:21.021965 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.021975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.034424 (+ 12449us) server_negotiation.cc:407] Connection header received
0620 20:15:21.035253 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.035261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.035349 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.035426 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.041895 (+ 6469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.043156 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.055118 (+ 11962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.056034 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.056199 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:21.066127 (+ 9928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:21.066140 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:21.066145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:21.066191 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:21.072167 (+ 5976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:21.073454 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:21.073491 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:21.073498 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:21.073792 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:21.074609 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:21.074617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:21.074623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:21.075527 (+ 904us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:21.077614 (+ 2087us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:21.077942 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 20:15:21.078566 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:15:21.091943 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:21.027698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.169.188:46715 (local address 127.0.0.1:46838)
0620 20:15:21.027911 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:21.027936 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:21.028082 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:21.037309 (+ 9227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:21.037318 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:21.037981 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:21.038268 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:21.038283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.042502 (+ 4219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.042511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:21.044542 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:21.044560 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.046938 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.046948 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:21.047314 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:21.048006 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:21.048039 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:21.052214 (+ 4175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:21.087227 (+ 35013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:21.087237 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:21.087244 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:21.088011 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:21.088957 (+ 946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:21.088965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:21.088971 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:21.089211 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:21.090647 (+ 1436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:21.090656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:21.090866 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 20:15:21.091042 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 20:15:21.091480 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 20:15:21.092372 27769 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [term 1 FOLLOWER]: Refusing update from remote peer ddef03c3a0df4b469a15c52b10c55069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:21.093709 27823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:21.094091 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:21.028025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.4.169.188:46715)
0620 20:15:21.036625 (+ 8600us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.036634 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.036693 (+ 59us) server_negotiation.cc:407] Connection header received
0620 20:15:21.036816 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.036825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.036979 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.038540 (+ 1561us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.039151 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.042192 (+ 3041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.044870 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:21.046656 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:21.047813 (+ 1157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:21.053001 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:21.053040 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:21.053050 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:21.053135 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:21.086899 (+ 33764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:21.088372 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:21.088382 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:21.088387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:21.088736 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:21.089433 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:21.089440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:21.089446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:21.090339 (+ 893us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:21.092263 (+ 1924us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:21.092475 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 20:15:21.092991 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8299,"thread_start_us":10129,"threads_started":1}
I20260620 20:15:21.099222 27840 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:15:21.101361 27720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7: Bootstrap starting.
I20260620 20:15:21.106500 27720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:21.108541 27720 log.cc:824] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:21.121289 27720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7: No bootstrap required, opened a new log
I20260620 20:15:21.103089 27579 master_runner.cc:186] Error getting master registration for 127.4.169.189:42391: OK, instance_id { permanent_uuid: "86740971fc184137a86d2946168911d7" instance_seqno: 1781986520300208 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:15:21.122426 27579 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:15:21.125365 27802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddef03c3a0df4b469a15c52b10c55069. Latest consensus state: current_term: 1 leader_uuid: "ddef03c3a0df4b469a15c52b10c55069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:21.125669 27802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:21.127280 27823 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddef03c3a0df4b469a15c52b10c55069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:21.127574 27823 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:21.127852 27841 mvcc.cc:205] Tried to move back new op lower bound from 7299016790372691968 to 7299016790055350272. Current Snapshot: MvccSnapshot[applied={T|T < 7299016790372691968}]
I20260620 20:15:21.129421 27802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddef03c3a0df4b469a15c52b10c55069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:21.129648 27802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:21.131157 27829 catalog_manager.cc:1347] Generated new cluster ID: e2a73f7fd5d14cdab567edd0af9d9042
I20260620 20:15:21.131248 27829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:21.133287 27720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } }
I20260620 20:15:21.133682 27720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86740971fc184137a86d2946168911d7, State: Initialized, Role: FOLLOWER
I20260620 20:15:21.134186 27720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } }
I20260620 20:15:21.141487 27846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:21.141902 27846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:21.143713 27720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:21.145220 27694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:21.150061 27694 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [term 1 FOLLOWER]: Refusing update from remote peer ddef03c3a0df4b469a15c52b10c55069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:21.150604 27695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddef03c3a0df4b469a15c52b10c55069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "86740971fc184137a86d2946168911d7"
I20260620 20:15:21.155323 27695 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddef03c3a0df4b469a15c52b10c55069 in term 1.
I20260620 20:15:21.156925 27823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:21.164276 27829 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:21.166046 27829 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:21.195204 27846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddef03c3a0df4b469a15c52b10c55069. Latest consensus state: current_term: 1 leader_uuid: "ddef03c3a0df4b469a15c52b10c55069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:21.195515 27846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:21.195784 27855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddef03c3a0df4b469a15c52b10c55069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:21.196152 27855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:21.196108 27823 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddef03c3a0df4b469a15c52b10c55069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddef03c3a0df4b469a15c52b10c55069" member_type: VOTER last_known_addr { host: "127.4.169.190" port: 42601 } } peers { permanent_uuid: "86740971fc184137a86d2946168911d7" member_type: VOTER last_known_addr { host: "127.4.169.189" port: 42391 } } peers { permanent_uuid: "0cd761c4e3664090a3b30e69ef7cc87b" member_type: VOTER last_known_addr { host: "127.4.169.188" port: 46715 } } }
I20260620 20:15:21.196362 27823 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:21.208520 27829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddef03c3a0df4b469a15c52b10c55069: Generated new TSK 0
I20260620 20:15:21.209816 27829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:15:21.222599 27865 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 20:15:21.282658 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.279104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.4.169.190:42601)
0620 20:15:21.279387 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.279397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.279952 (+ 555us) server_negotiation.cc:407] Connection header received
0620 20:15:21.280269 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.280275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.280365 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.280460 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.282245 (+ 1785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57012: BlockingRecv error: recv got EOF from 127.0.0.1:57012 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 20:15:21.285656 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.280331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.4.169.189:42391)
0620 20:15:21.280811 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.280821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.283621 (+ 2800us) server_negotiation.cc:407] Connection header received
0620 20:15:21.283911 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.283920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.284031 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.284120 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.285350 (+ 1230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55110: BlockingRecv error: recv got EOF from 127.0.0.1:55110 (error 108)
Metrics: {"server-negotiator.queue_time_us":287}
W20260620 20:15:21.288884 27833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.287106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.4.169.188:46715)
0620 20:15:21.287317 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.287323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.287507 (+ 184us) server_negotiation.cc:407] Connection header received
0620 20:15:21.287699 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.287705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.287788 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.287896 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.288662 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46846: BlockingRecv error: recv got EOF from 127.0.0.1:46846 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 20:15:21.295142 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.293428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.4.169.189:42391)
0620 20:15:21.293605 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.293611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.293642 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:15:21.293894 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.293898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.293990 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.294105 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.294878 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55112: BlockingRecv error: recv got EOF from 127.0.0.1:55112 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 20:15:21.304292 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.299539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.4.169.189:42391)
0620 20:15:21.299743 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.299750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.302916 (+ 3166us) server_negotiation.cc:407] Connection header received
0620 20:15:21.303181 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.303185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.303251 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.303358 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.304087 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55120: BlockingRecv error: recv got EOF from 127.0.0.1:55120 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 20:15:21.352325 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.342592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.4.169.189:42391)
0620 20:15:21.342788 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.342794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.350982 (+ 8188us) server_negotiation.cc:407] Connection header received
0620 20:15:21.351214 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.351219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.351292 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.351403 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.352138 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55132: BlockingRecv error: recv got EOF from 127.0.0.1:55132 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 20:15:21.362352 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.359983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.4.169.189:42391)
0620 20:15:21.360285 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.360294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.360879 (+ 585us) server_negotiation.cc:407] Connection header received
0620 20:15:21.361096 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.361102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.361217 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.361333 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.362122 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55140: BlockingRecv error: recv got EOF from 127.0.0.1:55140 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260620 20:15:21.376358 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.374050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.4.169.189:42391)
0620 20:15:21.374289 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.374299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.374888 (+ 589us) server_negotiation.cc:407] Connection header received
0620 20:15:21.375104 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.375109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.375234 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.375354 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.376132 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55152: BlockingRecv error: recv got EOF from 127.0.0.1:55152 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 20:15:21.397828 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.395636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.4.169.189:42391)
0620 20:15:21.395838 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.395848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.396384 (+ 536us) server_negotiation.cc:407] Connection header received
0620 20:15:21.396630 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.396635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.396719 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.396826 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.397592 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55166: BlockingRecv error: recv got EOF from 127.0.0.1:55166 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 20:15:21.435767 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.433507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.4.169.189:42391)
0620 20:15:21.433745 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.433754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.434301 (+ 547us) server_negotiation.cc:407] Connection header received
0620 20:15:21.434572 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.434577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.434651 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.434761 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.435500 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55180: BlockingRecv error: recv got EOF from 127.0.0.1:55180 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 20:15:21.508599 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.506691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.4.169.189:42391)
0620 20:15:21.506896 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.506904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.507298 (+ 394us) server_negotiation.cc:407] Connection header received
0620 20:15:21.507462 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.507468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.507570 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.507697 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.508381 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55186: BlockingRecv error: recv got EOF from 127.0.0.1:55186 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 20:15:21.645988 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.644021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.4.169.189:42391)
0620 20:15:21.644187 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.644192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.644625 (+ 433us) server_negotiation.cc:407] Connection header received
0620 20:15:21.644904 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.644908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.644994 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.645089 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.645776 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55188: BlockingRecv error: recv got EOF from 127.0.0.1:55188 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 20:15:21.907156 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:21.905463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.4.169.189:42391)
0620 20:15:21.905641 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:21.905645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:21.905938 (+ 293us) server_negotiation.cc:407] Connection header received
0620 20:15:21.906117 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:21.906121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:21.906190 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:21.906297 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:21.906972 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55196: BlockingRecv error: recv got EOF from 127.0.0.1:55196 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:15:21.911708 27813 catalog_manager.cc:1259] Loaded cluster ID: e2a73f7fd5d14cdab567edd0af9d9042
I20260620 20:15:21.911803 27813 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b: loading cluster ID for follower catalog manager: success
I20260620 20:15:21.914564 27813 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b: acquiring CA information for follower catalog manager: success
I20260620 20:15:21.916249 27813 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0cd761c4e3664090a3b30e69ef7cc87b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:15:22.227046 27865 catalog_manager.cc:1259] Loaded cluster ID: e2a73f7fd5d14cdab567edd0af9d9042
I20260620 20:15:22.227285 27865 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7: loading cluster ID for follower catalog manager: success
I20260620 20:15:22.230053 27865 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7: acquiring CA information for follower catalog manager: success
I20260620 20:15:22.231921 27865 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 86740971fc184137a86d2946168911d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:15:22.451695 27930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:22.432249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.4.169.189:42391)
0620 20:15:22.442908 (+ 10659us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:22.442916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:22.443303 (+ 387us) server_negotiation.cc:407] Connection header received
0620 20:15:22.443561 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:22.443567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:22.443788 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:22.443920 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:22.444626 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.446000 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.448507 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.449218 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.449356 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:22.450072 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:22.450912 (+ 840us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:22.451069 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:22.451222 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 20:15:22.451313 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10487,"thread_start_us":10300,"threads_started":1}
I20260620 20:15:22.484401 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:22.462315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.4.169.188:46715)
0620 20:15:22.478714 (+ 16399us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:22.478725 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:22.478742 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:15:22.478847 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:22.478850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:22.478998 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:22.479192 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:22.479636 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.480395 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.481773 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.482296 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.482461 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:22.483037 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:22.483659 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:22.483865 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:22.483963 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 20:15:22.484044 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16291,"thread_start_us":16186,"threads_started":1}
I20260620 20:15:22.511405 27930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:22.492002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.4.169.189:42391)
0620 20:15:22.492217 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:22.492223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:22.499019 (+ 6796us) server_negotiation.cc:407] Connection header received
0620 20:15:22.499333 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:22.499343 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:22.499432 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:22.499503 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:22.500544 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.501988 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.508404 (+ 6416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.509660 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.509775 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:22.510454 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:22.510585 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:22.510650 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:22.510902 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 20:15:22.510998 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:15:22.516809 27942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:22.491385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.4.169.190:42601)
0620 20:15:22.504847 (+ 13462us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:22.504856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:22.504880 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:15:22.505021 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:22.505029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:22.505282 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:22.506309 (+ 1027us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:22.506570 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.508166 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.513793 (+ 5627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.514501 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.515147 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:22.515345 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:22.515782 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:22.516316 (+ 534us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:22.516366 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 20:15:22.516449 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13299,"thread_start_us":13178,"threads_started":1}
I20260620 20:15:22.518932 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:22.499233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.4.169.188:46715)
0620 20:15:22.499844 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:22.499850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:22.510053 (+ 10203us) server_negotiation.cc:407] Connection header received
0620 20:15:22.510504 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:22.510509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:22.510610 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:22.510675 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:22.511417 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.512530 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.516201 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:22.516812 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:22.516919 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:22.518070 (+ 1151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:22.518210 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:22.518278 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:22.518529 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 20:15:22.518626 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
I20260620 20:15:22.522130 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 27579
I20260620 20:15:22.523625 27641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:22.634904 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 27579
I20260620 20:15:22.655475 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 27653
I20260620 20:15:22.656597 27715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:22.762012 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 27653
I20260620 20:15:22.781821 4774 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu pid 27727
I20260620 20:15:22.782920 27789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:22.892202 4774 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBzMLvU/build/asan/bin/kudu with pid 27727
2026-06-20T20:15:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4915 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9708 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (369948 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST